Sep 10 23:34:36.410585 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 23:34:36.410769 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 23:34:36.410852 ip-10-0-131-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 23:34:36.411464 ip-10-0-131-31 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 23:34:46.445785 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 23:34:46.445809 ip-10-0-131-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 49ad22da4bc9471cbdf0323ac5e1f1ba --
Sep 10 23:36:34.755202 ip-10-0-131-31 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:36:35.112245 ip-10-0-131-31 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:35.112245 ip-10-0-131-31 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:36:35.112245 ip-10-0-131-31 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:35.112245 ip-10-0-131-31 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:36:35.112245 ip-10-0-131-31 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:36:35.113148 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.113066    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:36:35.117690 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117675    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.117690 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117691    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117696    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117700    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117704    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117713    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117716    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117719    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117722    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117724    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117727    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117729    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117732    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117736    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117739    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117742    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117745    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117748    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117750    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117754    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.117753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117757    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117760    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117762    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117765    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117768    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117771    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117773    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117776    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117778    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117781    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117783    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117786    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117788    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117791    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117793    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117796    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117798    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117801    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117803    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117806    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.118229 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117808    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117811    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117813    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117816    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117819    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117821    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117823    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117826    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117828    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117831    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117833    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117836    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117838    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117840    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117843    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117846    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117849    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117851    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117854    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117856    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.118707 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117859    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117861    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117864    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117866    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117869    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117871    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117874    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117876    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117879    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117881    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117884    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117888    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117890    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117893    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117895    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117898    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117900    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117902    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117905    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.119557 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117907    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117910    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117912    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117915    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117917    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117919    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117922    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117943    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117947    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117950    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117953    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117955    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117958    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117960    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117963    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117966    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117968    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117971    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117973    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.117976    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.120292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119547    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119566    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119573    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119579    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119585    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119589    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119594    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119602    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119609    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119614    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119619    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119623    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119633    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119638    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119642    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119646    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119651    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119655    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119660    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.120906 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119664    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119668    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119673    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119677    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119681    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119686    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119695    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119699    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119703    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119707    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119711    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119716    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119720    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119725    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119731    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119736    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119740    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119745    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119754    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.121435 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119758    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119763    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119768    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119772    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119776    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119782    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119787    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119791    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119795    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119799    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119805    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119815    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119819    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119823    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119828    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119832    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119837    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119841    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119846    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119850    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.121903 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119854    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119858    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119862    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119867    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119876    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119880    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119885    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119889    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119895    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119899    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119903    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119908    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119913    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119917    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119921    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119947    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119957    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119961    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119968    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119972    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.122643 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119976    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119980    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119984    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119988    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119992    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.119997    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120001    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120005    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120013    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120018    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120022    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120026    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120031    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120043    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120047    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120051    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120056    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120060    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120065    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120068    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.123371 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120074    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120783    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120795    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120798    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120801    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120804    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120808    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120811    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120813    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120816    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120819    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120821    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120824    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120826    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120829    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120831    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120834    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120837    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120839    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120842    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.123867 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120844    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120847    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120849    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120851    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120854    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120856    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120859    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120862    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120864    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120867    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120869    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120871    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120875    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120880    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120883    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120885    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120888    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120891    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120893    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120901    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.124379 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120903    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120906    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120909    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120911    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120914    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120916    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120919    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120923    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120947    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120952    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120955    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120958    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120961    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120963    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120966    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120968    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120971    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120973    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120976    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.124873 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120978    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120981    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120983    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120986    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120988    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120991    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.120998    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121002    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121005    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121008    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121012    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121014    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121017    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121020    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121023    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121025    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121027    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121030    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121032    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.125394 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121035    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121039    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121041    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121044    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121046    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121051    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121054    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121057    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121060    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121063    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121066    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121069    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121072    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121074    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121077    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121079    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121082    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121084    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121087    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.125917 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121090    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121093    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121095    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121165    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121200    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121207    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121211    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121216    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121219    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121223    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121227    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121230    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121233    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121236    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121239    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121242    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121245    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121248    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121251    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121255    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121258    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121263    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121266    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121270    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:36:35.126425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121273    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121277    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121281    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121283    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121287    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121290    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121293    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121296    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121300    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121304    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121307    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121315    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121318    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121321    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121324    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121327    2574 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121330    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121334    2574 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121337    2574 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121340    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121343    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121346    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121349    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121352    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121355    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:36:35.127025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121358    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121360    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121363    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121367    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121369    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121372    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121375    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121378    2574 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121382    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121385    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121388    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121391    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121394    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121397    2574 flags.go:64] FLAG: --help="false"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121400    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121404    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121407    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121410    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121413    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121416    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121419    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121422    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121425    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:36:35.127640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121428    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121431    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121434    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121437    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121440    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121443    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121446    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121449    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121452    2574 flags.go:64] FLAG: --lock-file=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121455    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121458    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121461    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121466    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121469    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121472    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121475    2574 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121478    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121481    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121484    2574 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121487    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121490    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121493    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121497    2574 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121500    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121503    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:36:35.128315 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121506    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121509    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121512    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121515    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121518    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121524    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121527    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121530    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121533    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121536    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121541    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121544    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121547    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121550    2574 flags.go:64] FLAG: --port="10250"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121553    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121556    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a8e823b42649146"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121559    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121561    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121564    2574 flags.go:64] FLAG: --register-node="true"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121567    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121571    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121574    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121579    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121582    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121586    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:36:35.128949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121590    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121593    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121595    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121598    2574 flags.go:64] FLAG: --runonce="false"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121601    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121604    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121607    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121610    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121613    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121616    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121619    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121622    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121625    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121628    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121631    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121634    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121637    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121640    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121642    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121648    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121650    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121653    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121660    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121663    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121665    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:36:35.129597 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121668    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121671    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121674    2574 flags.go:64] FLAG: --v="2"
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121678    2574 flags.go:64] FLAG: --version="false"
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121685    2574 flags.go:64] FLAG: --vmodule=""
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121689    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.121692    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121806    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121810    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121813    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121816    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121819    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121822    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121826    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121828    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121831    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121833    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121837    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121839    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121842    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121844    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.130227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121848    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121852    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121855    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121858    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121861    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121864    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121866    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121869    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121871    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121874    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121876    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121879    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121881    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121884    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121887    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121891    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121894    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121896    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121899    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121901    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.130737 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121904    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121907    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121910    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121913    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121915    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121918    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121921    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.121923    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122005    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122009    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122012    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122015    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122017    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122020    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122023    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122025    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122028    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122031    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122033    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122036    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.131313 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122038    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122041    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122043    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122048    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122050    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122053    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122056    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122059    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122063    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122067    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122069    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122072    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122074    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122077    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122080    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122082    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122084    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122089    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122091    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.131798 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122094    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122096    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122099    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122101    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122104    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122107    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122109    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122112    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122114    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122117    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122120    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122123    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122125    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122128    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122130    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122133    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122138    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122141    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122144    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.132282 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122147    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122150    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122153    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122156    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122159    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122161    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.122164    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.122170    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.129288    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.129302    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.129987    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.129995    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130062    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130068    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.132780 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130072    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130075    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130079    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130082    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130084    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130088    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130091    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130093    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130096    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130098    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130101    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130104    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130107    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130109    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130112    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130114    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130117    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130120    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130123    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130126    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.133145 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130129    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130131    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130134    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130137    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130140    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130142    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130146    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130149    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130152    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130155    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130158    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130160    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130163    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130166    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130168    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130171    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130174    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130176    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130179    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130182    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.133630 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130184    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130187    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130190    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130193    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130196    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130198    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130201    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130204    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130206    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130209    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130211    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130214    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130216    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130219    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130222    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130226    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130229    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130233    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130236    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130239    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.134132 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130246    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130249    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130252    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130255    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130258    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130260    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130263    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130266    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130268    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130271    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130275    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130278    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130281    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130284    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130286    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130289    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130292    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130295    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130298    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.134614 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130301    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130303    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130306    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130309    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130312    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130314    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130318    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130320    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130323    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130325    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130329    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130331    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130334    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130337    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130340    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130342    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130345    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.135155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130348    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.130353    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130461    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130465    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130468    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130471    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130474    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130476    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130479    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130482    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130485    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130487    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130491    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130495    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:36:35.135577 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130497    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130501    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130503    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130506    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130509    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130512    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130515    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130518    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130522    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130525    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130528    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130531    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130534    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130537    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130540    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130542    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130545    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130548    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130550    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130553    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:36:35.135940 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130556    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130559    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130561    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130564    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130566    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130569    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130571    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130574    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130577    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130579    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130582    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130585    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130587    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130590    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130592    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130595    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130597    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130600    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130602    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130605    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:36:35.136405 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130608    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130610    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130613    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130616    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130619    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130621    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130624    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130626    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130629    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130631    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130634    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130636    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130639    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130641    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130644    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130647    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130649    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130652    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130655    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130658    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:36:35.136889 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130661    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130664    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130667    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130670    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130672    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130675    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130678    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130681    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130684    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130687    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130690    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130695    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130698    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130700    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130703    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130706    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130709    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130711    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130714    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130716    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:36:35.137380 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130719    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130722    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130725    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130728    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130730    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130733    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:35.130736    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.130741    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.130854    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.134562    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.135529    2574 server.go:1019] "Starting client certificate rotation"
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.135627    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.135676    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:36:35.137861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.137609    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:36:35.139972 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.139957    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:36:35.153443 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.153422    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:36:35.156042 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.156027    2574 log.go:25] "Validated CRI v1 image API"
Sep 10 23:36:35.157887 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.157870    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:36:35.160745 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.160728    2574 fs.go:135] Filesystem UUIDs: map[44444aef-8807-449e-abeb-1f31c9f9ea4b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e9d79d32-9375-4e61-b1f6-b304e6d7364a:/dev/nvme0n1p4]
Sep 10 23:36:35.160839 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.160745    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:36:35.161075 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.161059    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:36:35.165626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.165520    2574 manager.go:216] Machine: {Timestamp:2026-09-10 23:36:35.164293824 +0000 UTC m=+0.301958291 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101769 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ef0e6f33d631efb9c2e8b8b0422c SystemUUID:ec28ef0e-6f33-d631-efb9-c2e8b8b0422c BootID:49ad22da-4bc9-471c-bdf0-323ac5e1f1ba Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8f:a4:4d:aa:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8f:a4:4d:aa:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:1f:a6:e9:89:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:36:35.165626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.165620    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:36:35.165736 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.165699    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:36:35.168863 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.168838    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:36:35.169014 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.168865    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:36:35.171145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.171135    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:36:35.171177 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.171149    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:36:35.171177 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.171157    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:36:35.171803 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.171793    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:36:35.172986 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.172976    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:36:35.176096 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176085    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:36:35.176203 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176194    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:36:35.176236 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176223    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:36:35.176263 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176240    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:36:35.176263 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176254    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:36:35.176263 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176263    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:36:35.176385 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176274    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:36:35.176606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176595    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:36:35.176636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.176615    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:36:35.179030 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.179013    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:36:35.181734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.181721    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:36:35.181781 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.181763    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:36:35.183445 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183434    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:36:35.183482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183450    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:36:35.183482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183456    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:36:35.183482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183461    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:36:35.183482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183472    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:36:35.183482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183481    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183497    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183502    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183508    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183513    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183530    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:36:35.183606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.183536    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:36:35.184303 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.184294    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:36:35.184303 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.184303    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:36:35.185119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.185099    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjlzf"
Sep 10 23:36:35.186989 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.186976    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:36:35.187074 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.187056    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:36:35.187235 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.187219    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:36:35.188066 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.188053    2574 server.go:1268] "Started kubelet"
Sep 10 23:36:35.188164 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.188135    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:36:35.188662 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.188612    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:36:35.188755 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.188681    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:36:35.188972 ip-10-0-131-31 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:36:35.192725 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.192583    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjlzf"
Sep 10 23:36:35.192801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.192632    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:36:35.194174 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.194147    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:36:35.197627 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.197608    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:36:35.197707 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.195866    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-31.ec2.internal.18d419de73bb6924  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-31.ec2.internal,UID:ip-10-0-131-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-31.ec2.internal,},FirstTimestamp:2026-09-10 23:36:35.188025636 +0000 UTC m=+0.325690102,LastTimestamp:2026-09-10 23:36:35.188025636 +0000 UTC m=+0.325690102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-31.ec2.internal,}"
Sep 10 23:36:35.201337 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.201317    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:36:35.201916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.201901    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:36:35.202050 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.202031    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.202686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202667    2574 factory.go:55] Registering systemd factory
Sep 10 23:36:35.202686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202672    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:36:35.202813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202699    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:36:35.202813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202671    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:36:35.202813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202687    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:36:35.202941 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202823    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:36:35.202941 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202831    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:36:35.203018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.202992    2574 factory.go:153] Registering CRI-O factory
Sep 10 23:36:35.203018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.203001    2574 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:36:35.203074 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.203038    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:36:35.203074 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.203054    2574 factory.go:103] Registering Raw factory
Sep 10 23:36:35.203074 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.203065    2574 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:36:35.203439 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.203426    2574 manager.go:349] Starting recovery of all containers
Sep 10 23:36:35.214370 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.214211    2574 manager.go:354] Recovery completed
Sep 10 23:36:35.215852 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.215831    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:35.218322 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.218299    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-31.ec2.internal\" not found" node="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.219876 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.219863    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.221862 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.221847    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.221921 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.221877    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.221921 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.221888    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.222459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.222445    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:36:35.222459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.222458    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:36:35.222570 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.222478    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:36:35.222570 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.222463    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.223287 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223273    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.223383 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223296    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.223383 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223316    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.223627 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223615    2574 policy_none.go:49] "None policy: Start"
Sep 10 23:36:35.223673 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223631    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:36:35.223673 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.223640    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:36:35.225050 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.225039    2574 policy_none.go:47] "Start"
Sep 10 23:36:35.253589 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.253574    2574 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.253607    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.253619    2574 server.go:87] "Starting device plugin registration server"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.253900    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.253910    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.254020    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.254099    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.254108    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.254593    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:36:35.280769 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.254639    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.298945 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.298916    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:36:35.300016 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.300002    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:36:35.300080 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.300042    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:36:35.300080 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.300061    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:36:35.300165 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.300094    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:36:35.302532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.302513    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:35.354582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.354571    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.355384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.355368    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.355448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.355403    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.355448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.355417    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.355448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.355445    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.363275 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.363221    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.363275 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.363244    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-31.ec2.internal\": node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.390217 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.390198    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.400443 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.400427    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"]
Sep 10 23:36:35.400494 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.400487    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.404256 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.404235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.404348 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.404269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.404401 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.404389    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.404442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.404416    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.404442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.404429    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.406552 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.406540    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.406686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.406672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.406726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.406702    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.407310 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407289    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.407382 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407322    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.407382 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407339    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.407488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407381    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.407488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407399    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.407488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.407408    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.409534 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.409517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.409614 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.409548    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:36:35.410339 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.410325    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:36:35.410427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.410349    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:36:35.410427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.410359    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:36:35.437469 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.437451    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-31.ec2.internal\" not found" node="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.441792 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.441774    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-31.ec2.internal\" not found" node="ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.490993 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.490978    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.504499 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.504484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.504572 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.504507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.504572 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.504527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9761818410ff00337ec1726a7473d4-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"6e9761818410ff00337ec1726a7473d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.504640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.504591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.504640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.504597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fabeef8f745298115c8060203460ffaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"fabeef8f745298115c8060203460ffaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.591718 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.591693    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.605053 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.605032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9761818410ff00337ec1726a7473d4-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"6e9761818410ff00337ec1726a7473d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.605120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.605085    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9761818410ff00337ec1726a7473d4-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"6e9761818410ff00337ec1726a7473d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.692519 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.692432    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.740895 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.740865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.745697 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:35.745513    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:35.792540 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.792508    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.893000 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.892966    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:35.993496 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:35.993434    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:37.401823 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:37.401799    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:37.443237 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.443216    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:36:37.443665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.443355    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:36:37.443665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.443407    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:36:37.475435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.475411    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:37.502005 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:37.501978    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found"
Sep 10 23:36:37.509056 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.509022    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:31:36 +0000 UTC" deadline="2028-05-10 01:59:12.322639939 +0000 UTC"
Sep 10 23:36:37.509056 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.509053    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14570h22m34.813589515s"
Sep 10 23:36:37.536032 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.536013    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:37.610500 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.610413    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:37.612194 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:37.612165    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabeef8f745298115c8060203460ffaa.slice/crio-aa8ec0e7b75d1bd6fa072e4e4b175eb949ea2e85f7241c37e1ba962c80e86a62 WatchSource:0}: Error finding container aa8ec0e7b75d1bd6fa072e4e4b175eb949ea2e85f7241c37e1ba962c80e86a62: Status 404 returned error can't find the container with id aa8ec0e7b75d1bd6fa072e4e4b175eb949ea2e85f7241c37e1ba962c80e86a62
Sep 10 23:36:37.612511 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:37.612491    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9761818410ff00337ec1726a7473d4.slice/crio-43f3a9cf25ae7a10375fb4b3169d2fa367d924f83a56f088fc1c940837846050 WatchSource:0}: Error finding container 43f3a9cf25ae7a10375fb4b3169d2fa367d924f83a56f088fc1c940837846050: Status 404 returned error can't find the container with id 43f3a9cf25ae7a10375fb4b3169d2fa367d924f83a56f088fc1c940837846050
Sep 10 23:36:37.616966 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.616949    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:36:37.620718 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.620699    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:36:37.621437 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.621423    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal"
Sep 10 23:36:37.628191 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:37.628175    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:36:38.486192 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.486161    2574 apiserver.go:52] "Watching apiserver"
Sep 10 23:36:38.492958 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.492932    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:36:38.493340 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.493317    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dr968","openshift-dns/node-resolver-z8xph","openshift-multus/multus-6rmhq","openshift-multus/multus-additional-cni-plugins-frwgx","openshift-image-registry/node-ca-4r6t6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal","openshift-multus/network-metrics-daemon-msf67","openshift-network-diagnostics/network-check-target-h28ql","openshift-network-operator/iptables-alerter-jwps4","openshift-ovn-kubernetes/ovnkube-node-6hknm","kube-system/global-pull-secret-syncer-528jm","kube-system/konnectivity-agent-mqz5d","kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"]
Sep 10 23:36:38.495828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.495810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.495913 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.495889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:38.497997 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.497967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.498082 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.498055    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:38.500197 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.500161    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.502358 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.502335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.502728 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.502708    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:36:38.502829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.502785    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:36:38.502889 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.502827    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.503027 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.503003    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.503094 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.503074    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cj4dq\""
Sep 10 23:36:38.504251 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.504232    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:36:38.504354 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.504274    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x9st2\""
Sep 10 23:36:38.504354 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.504295    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:36:38.505327 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.505310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.507332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.507222    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:36:38.507332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.507221    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.507453 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.507337    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wdj8v\""
Sep 10 23:36:38.507506 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.507462    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.507715 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.507698    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.509471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.509452    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkhdv\""
Sep 10 23:36:38.509564 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.509524    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:36:38.509625 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.509584    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.509706 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.509693    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.510055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.510007    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.510134 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.510110    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:36:38.512033 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.512014    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.512115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.512022    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6n45x\""
Sep 10 23:36:38.512229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.512210    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:36:38.512308 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.512279    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.512308 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.512292    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:38.512422 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.512400    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:38.515293 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.515261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.517368 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.517170    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwmdk\""
Sep 10 23:36:38.517368 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.517319    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.517513 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.517455    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.518238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.518219    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:36:38.519039 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.519020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.520003 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.519986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-dbus\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.520090 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjp4\" (UniqueName: \"kubernetes.io/projected/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kube-api-access-chjp4\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.520090 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-k8s-cni-cncf-io\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520090 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-netns\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-socket-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cnibin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cni-binary-copy\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.520248 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab405fb-bc66-4ffa-9116-49881c27576a-host\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-sys-fs\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqvq\" (UniqueName: \"kubernetes.io/projected/827a263b-611a-4335-8db3-1d4a4009fda2-kube-api-access-zgqvq\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-multus\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-multus-certs\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-kubelet-config\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/827a263b-611a-4335-8db3-1d4a4009fda2-iptables-alerter-script\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-kubelet\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520566    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-conf-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.520621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-daemon-config\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520629    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cnibin\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-registration-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520682    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-system-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-socket-dir-parent\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-hostroot\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdwf\" (UniqueName: \"kubernetes.io/projected/d567f9a9-22fe-4fc9-82a4-a04294e982e0-kube-api-access-2xdwf\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkc7q\" (UniqueName: \"kubernetes.io/projected/906f16ef-b837-4aac-b339-f93860a0053a-kube-api-access-hkc7q\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-device-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.520990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529kq\" (UniqueName: \"kubernetes.io/projected/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-kube-api-access-529kq\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab405fb-bc66-4ffa-9116-49881c27576a-serviceca\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827a263b-611a-4335-8db3-1d4a4009fda2-host-slash\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.521145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521139    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-bin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521170    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t5ldc\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-etc-kubernetes\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4tlj\" (UniqueName: \"kubernetes.io/projected/6ab405fb-bc66-4ffa-9116-49881c27576a-kube-api-access-w4tlj\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521252    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521383    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521395    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521528    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521620    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:36:38.521965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.521694    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.523376 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.523348    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:36:38.523639 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.523591    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hb9kx\""
Sep 10 23:36:38.523739 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.523719    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:36:38.523856 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.523839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.525743 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.525727    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2hpfs\""
Sep 10 23:36:38.525894 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.525759    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:36:38.525894 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.525760    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:36:38.537119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.537103    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hdbr4"
Sep 10 23:36:38.544404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.544388    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hdbr4"
Sep 10 23:36:38.612675 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.612645    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:36:38.614125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.614048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"fabeef8f745298115c8060203460ffaa","Type":"ContainerStarted","Data":"aa8ec0e7b75d1bd6fa072e4e4b175eb949ea2e85f7241c37e1ba962c80e86a62"}
Sep 10 23:36:38.615325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.615286    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" event={"ID":"6e9761818410ff00337ec1726a7473d4","Type":"ContainerStarted","Data":"43f3a9cf25ae7a10375fb4b3169d2fa367d924f83a56f088fc1c940837846050"}
Sep 10 23:36:38.621733 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqvq\" (UniqueName: \"kubernetes.io/projected/827a263b-611a-4335-8db3-1d4a4009fda2-kube-api-access-zgqvq\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-multus\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-multus-certs\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-config\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-kubelet\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621838    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-multus\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.621844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-daemon-config\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cnibin\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cnibin\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-host\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-kubelet\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-multus-certs\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.621976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-registration-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-system-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-registration-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-hostroot\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-script-lib\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c599127-22a5-4cac-9bb0-48fbc8865853-konnectivity-ca\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.622120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab405fb-bc66-4ffa-9116-49881c27576a-serviceca\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-system-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5g8t\" (UniqueName: \"kubernetes.io/projected/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-kube-api-access-j5g8t\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-hostroot\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827a263b-611a-4335-8db3-1d4a4009fda2-host-slash\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-bin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-etc-kubernetes\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.622328    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4tlj\" (UniqueName: \"kubernetes.io/projected/6ab405fb-bc66-4ffa-9116-49881c27576a-kube-api-access-w4tlj\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-tmp-dir\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827a263b-611a-4335-8db3-1d4a4009fda2-host-slash\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622382    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-daemon-config\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.622391    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:39.122369607 +0000 UTC m=+2.952196825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-etc-kubernetes\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-node-log\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.622872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622470    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-var-lib-cni-bin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-k8s-cni-cncf-io\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-bin\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab405fb-bc66-4ffa-9116-49881c27576a-serviceca\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-run\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-var-lib-kubelet\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-k8s-cni-cncf-io\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c599127-22a5-4cac-9bb0-48fbc8865853-agent-certs\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cnibin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9sl\" (UniqueName: \"kubernetes.io/projected/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-kube-api-access-9w9sl\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622785    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-kubelet\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.623682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-slash\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cnibin\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.622953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-log-socket\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovn-node-metrics-cert\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-sys-fs\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-netns\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-tmp\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623297    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-sys-fs\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-kubelet-config\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/827a263b-611a-4335-8db3-1d4a4009fda2-iptables-alerter-script\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-conf-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-etc-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623416    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-kubelet-config\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.624404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-env-overrides\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623441    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-conf-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-kubernetes\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-socket-dir-parent\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdwf\" (UniqueName: \"kubernetes.io/projected/d567f9a9-22fe-4fc9-82a4-a04294e982e0-kube-api-access-2xdwf\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623566    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-hosts-file\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-conf\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-cni-dir\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-etc-tuned\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-multus-socket-dir-parent\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623652    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkc7q\" (UniqueName: \"kubernetes.io/projected/906f16ef-b837-4aac-b339-f93860a0053a-kube-api-access-hkc7q\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-device-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.625021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529kq\" (UniqueName: \"kubernetes.io/projected/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-kube-api-access-529kq\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-systemd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-ovn\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-netd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/827a263b-611a-4335-8db3-1d4a4009fda2-iptables-alerter-script\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysconfig\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-device-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-dbus\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.623989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjp4\" (UniqueName: \"kubernetes.io/projected/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kube-api-access-chjp4\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-netns\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624046    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53618be8-c85c-4f61-acb4-1e738be631f0-dbus\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-systemd-units\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d567f9a9-22fe-4fc9-82a4-a04294e982e0-host-run-netns\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-var-lib-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-modprobe-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.625557 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-systemd\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624294    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztf9\" (UniqueName: \"kubernetes.io/projected/ad580eec-bd50-4887-a786-5c9d6971872f-kube-api-access-xztf9\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-socket-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cni-binary-copy\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-sys\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-lib-modules\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.624413    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab405fb-bc66-4ffa-9116-49881c27576a-host\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.624478    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:36:39.124463823 +0000 UTC m=+2.954291024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624492    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab405fb-bc66-4ffa-9116-49881c27576a-host\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624595    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522b5148-f95b-4ec5-aa9d-59f06fe242e8-socket-dir\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.626172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.624912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d567f9a9-22fe-4fc9-82a4-a04294e982e0-cni-binary-copy\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.629925 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.629904    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:38.630014 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.629930    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:38.630014 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.629944    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:38.630014 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:38.630011    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:36:39.129997177 +0000 UTC m=+2.959824397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:38.630457 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.630430    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:36:38.635214 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdwf\" (UniqueName: \"kubernetes.io/projected/d567f9a9-22fe-4fc9-82a4-a04294e982e0-kube-api-access-2xdwf\") pod \"multus-6rmhq\" (UID: \"d567f9a9-22fe-4fc9-82a4-a04294e982e0\") " pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.635325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635085    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529kq\" (UniqueName: \"kubernetes.io/projected/5ae77854-f07b-45ff-8684-69dc0ae9d1b2-kube-api-access-529kq\") pod \"multus-additional-cni-plugins-frwgx\" (UID: \"5ae77854-f07b-45ff-8684-69dc0ae9d1b2\") " pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.635532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4tlj\" (UniqueName: \"kubernetes.io/projected/6ab405fb-bc66-4ffa-9116-49881c27576a-kube-api-access-w4tlj\") pod \"node-ca-4r6t6\" (UID: \"6ab405fb-bc66-4ffa-9116-49881c27576a\") " pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.635532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjp4\" (UniqueName: \"kubernetes.io/projected/522b5148-f95b-4ec5-aa9d-59f06fe242e8-kube-api-access-chjp4\") pod \"aws-ebs-csi-driver-node-g9xhd\" (UID: \"522b5148-f95b-4ec5-aa9d-59f06fe242e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.635653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635598    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqvq\" (UniqueName: \"kubernetes.io/projected/827a263b-611a-4335-8db3-1d4a4009fda2-kube-api-access-zgqvq\") pod \"iptables-alerter-jwps4\" (UID: \"827a263b-611a-4335-8db3-1d4a4009fda2\") " pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.635653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.635642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkc7q\" (UniqueName: \"kubernetes.io/projected/906f16ef-b837-4aac-b339-f93860a0053a-kube-api-access-hkc7q\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:38.670559 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.670537    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:38.725473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-hosts-file\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.725641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-conf\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.725641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-etc-tuned\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.725641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-hosts-file\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.725641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-systemd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-ovn\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-netd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-conf\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-systemd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysconfig\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-systemd-units\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-ovn\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysconfig\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-var-lib-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725831    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-systemd-units\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-netd\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.725849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-modprobe-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-var-lib-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-systemd\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztf9\" (UniqueName: \"kubernetes.io/projected/ad580eec-bd50-4887-a786-5c9d6971872f-kube-api-access-xztf9\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-modprobe-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.725952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-sys\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-sys\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726141    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-systemd\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-lib-modules\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-config\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-lib-modules\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-host\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726327    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-run-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-script-lib\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c599127-22a5-4cac-9bb0-48fbc8865853-konnectivity-ca\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726374    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-host\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5g8t\" (UniqueName: \"kubernetes.io/projected/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-kube-api-access-j5g8t\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-tmp-dir\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-node-log\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-bin\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-run\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-var-lib-kubelet\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c599127-22a5-4cac-9bb0-48fbc8865853-agent-certs\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.726661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9sl\" (UniqueName: \"kubernetes.io/projected/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-kube-api-access-9w9sl\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-kubelet\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-slash\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-log-socket\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-config\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovn-node-metrics-cert\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-cni-bin\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-netns\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovnkube-script-lib\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-tmp\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.726980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-etc-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-env-overrides\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-kubernetes\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-kubernetes\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-kubelet\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727217    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-tmp-dir\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.727221 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-slash\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727231    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-etc-openvswitch\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-node-log\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-run\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-log-socket\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727476    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-etc-sysctl-d\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c599127-22a5-4cac-9bb0-48fbc8865853-konnectivity-ca\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad580eec-bd50-4887-a786-5c9d6971872f-var-lib-kubelet\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-host-run-netns\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.727983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.727636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-env-overrides\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.728543 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.728061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-etc-tuned\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.729460 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.729423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c599127-22a5-4cac-9bb0-48fbc8865853-agent-certs\") pod \"konnectivity-agent-mqz5d\" (UID: \"1c599127-22a5-4cac-9bb0-48fbc8865853\") " pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:38.729653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.729629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad580eec-bd50-4887-a786-5c9d6971872f-tmp\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.729783 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.729725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-ovn-node-metrics-cert\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.734338 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.734300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztf9\" (UniqueName: \"kubernetes.io/projected/ad580eec-bd50-4887-a786-5c9d6971872f-kube-api-access-xztf9\") pod \"tuned-dr968\" (UID: \"ad580eec-bd50-4887-a786-5c9d6971872f\") " pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.734532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.734509    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5g8t\" (UniqueName: \"kubernetes.io/projected/fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9-kube-api-access-j5g8t\") pod \"ovnkube-node-6hknm\" (UID: \"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.736227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.736146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9sl\" (UniqueName: \"kubernetes.io/projected/83a76ea4-06b0-4a7c-9b1f-aa1300d006f1-kube-api-access-9w9sl\") pod \"node-resolver-z8xph\" (UID: \"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1\") " pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.810986 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.810954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rmhq"
Sep 10 23:36:38.819920 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.819900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frwgx"
Sep 10 23:36:38.828437 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.828416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4r6t6"
Sep 10 23:36:38.828686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.828671    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:36:38.834386 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.834359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd"
Sep 10 23:36:38.843018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.842992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jwps4"
Sep 10 23:36:38.849604 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.849584    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8xph"
Sep 10 23:36:38.856243 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.856222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:36:38.862853 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.862837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr968"
Sep 10 23:36:38.869403 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:38.869387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:36:39.129281 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.129213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:39.129436 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.129276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:39.129436 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.129374    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:39.129436 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.129401    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:39.129567 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.129463    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:36:40.129422182 +0000 UTC m=+3.959249395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:39.129567 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.129490    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:40.129479629 +0000 UTC m=+3.959306827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:39.229753 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.229718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:39.229933 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.229900    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:39.229933 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.229924    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:39.229933 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.229935    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:39.230060 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:39.229985    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:36:40.229970511 +0000 UTC m=+4.059797712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:39.453607 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.453574    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b5148_f95b_4ec5_aa9d_59f06fe242e8.slice/crio-286acf4c3f297f22152a9bcfa7c36657c46f72496f763cb43a8882274cf4dd3a WatchSource:0}: Error finding container 286acf4c3f297f22152a9bcfa7c36657c46f72496f763cb43a8882274cf4dd3a: Status 404 returned error can't find the container with id 286acf4c3f297f22152a9bcfa7c36657c46f72496f763cb43a8882274cf4dd3a
Sep 10 23:36:39.461166 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.461144    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c599127_22a5_4cac_9bb0_48fbc8865853.slice/crio-863a57d8057d2e3d9b5d54066b5ada6f8896f502f9d6ea1d49aca12d8880aa21 WatchSource:0}: Error finding container 863a57d8057d2e3d9b5d54066b5ada6f8896f502f9d6ea1d49aca12d8880aa21: Status 404 returned error can't find the container with id 863a57d8057d2e3d9b5d54066b5ada6f8896f502f9d6ea1d49aca12d8880aa21
Sep 10 23:36:39.462696 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.462676    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae77854_f07b_45ff_8684_69dc0ae9d1b2.slice/crio-77e7d143d751f8efc1b334c49b26e9802940bcb1f11a7d842fecd1cd4a2dfda6 WatchSource:0}: Error finding container 77e7d143d751f8efc1b334c49b26e9802940bcb1f11a7d842fecd1cd4a2dfda6: Status 404 returned error can't find the container with id 77e7d143d751f8efc1b334c49b26e9802940bcb1f11a7d842fecd1cd4a2dfda6
Sep 10 23:36:39.463734 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.463676    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2bcda4_346d_4bc5_9b7c_c37d9ee029f9.slice/crio-089bc3a373d5fdd9bc085012ff247f04d065e8b240d63ed9a8fdd2c28fd09085 WatchSource:0}: Error finding container 089bc3a373d5fdd9bc085012ff247f04d065e8b240d63ed9a8fdd2c28fd09085: Status 404 returned error can't find the container with id 089bc3a373d5fdd9bc085012ff247f04d065e8b240d63ed9a8fdd2c28fd09085
Sep 10 23:36:39.464270 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.464249    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad580eec_bd50_4887_a786_5c9d6971872f.slice/crio-f51a0ec230deffb0cb71e092807f5fdc45c312dd7bc5142b9ec5650778ae5af6 WatchSource:0}: Error finding container f51a0ec230deffb0cb71e092807f5fdc45c312dd7bc5142b9ec5650778ae5af6: Status 404 returned error can't find the container with id f51a0ec230deffb0cb71e092807f5fdc45c312dd7bc5142b9ec5650778ae5af6
Sep 10 23:36:39.485618 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.485596    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827a263b_611a_4335_8db3_1d4a4009fda2.slice/crio-5a9344df81905e2352e0a1e9567a48532a4030b5cdbbd2fbe8c1342cf829729c WatchSource:0}: Error finding container 5a9344df81905e2352e0a1e9567a48532a4030b5cdbbd2fbe8c1342cf829729c: Status 404 returned error can't find the container with id 5a9344df81905e2352e0a1e9567a48532a4030b5cdbbd2fbe8c1342cf829729c
Sep 10 23:36:39.486431 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.486403    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a76ea4_06b0_4a7c_9b1f_aa1300d006f1.slice/crio-fb62a6bff95e632154abe6da6192870073c1239202aeb5026d95dee458c0d4b7 WatchSource:0}: Error finding container fb62a6bff95e632154abe6da6192870073c1239202aeb5026d95dee458c0d4b7: Status 404 returned error can't find the container with id fb62a6bff95e632154abe6da6192870073c1239202aeb5026d95dee458c0d4b7
Sep 10 23:36:39.487290 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.487268    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd567f9a9_22fe_4fc9_82a4_a04294e982e0.slice/crio-2244dd23f2affca7f0c220ae1218258102ea0981619343ba3b71e58d90dbbee4 WatchSource:0}: Error finding container 2244dd23f2affca7f0c220ae1218258102ea0981619343ba3b71e58d90dbbee4: Status 404 returned error can't find the container with id 2244dd23f2affca7f0c220ae1218258102ea0981619343ba3b71e58d90dbbee4
Sep 10 23:36:39.488311 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:36:39.488287    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab405fb_bc66_4ffa_9116_49881c27576a.slice/crio-be415a2d19c3ab110a66eccb8a76d175befc2161866d05cdbe82fe15f617c5d6 WatchSource:0}: Error finding container be415a2d19c3ab110a66eccb8a76d175befc2161866d05cdbe82fe15f617c5d6: Status 404 returned error can't find the container with id be415a2d19c3ab110a66eccb8a76d175befc2161866d05cdbe82fe15f617c5d6
Sep 10 23:36:39.547660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.547617    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:38 +0000 UTC" deadline="2028-06-17 06:41:53.320719866 +0000 UTC"
Sep 10 23:36:39.547660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.547660    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15487h5m13.773063431s"
Sep 10 23:36:39.619717 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.619679    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqz5d" event={"ID":"1c599127-22a5-4cac-9bb0-48fbc8865853","Type":"ContainerStarted","Data":"863a57d8057d2e3d9b5d54066b5ada6f8896f502f9d6ea1d49aca12d8880aa21"}
Sep 10 23:36:39.620920 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.620895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd" event={"ID":"522b5148-f95b-4ec5-aa9d-59f06fe242e8","Type":"ContainerStarted","Data":"286acf4c3f297f22152a9bcfa7c36657c46f72496f763cb43a8882274cf4dd3a"}
Sep 10 23:36:39.623194 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.623161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4r6t6" event={"ID":"6ab405fb-bc66-4ffa-9116-49881c27576a","Type":"ContainerStarted","Data":"be415a2d19c3ab110a66eccb8a76d175befc2161866d05cdbe82fe15f617c5d6"}
Sep 10 23:36:39.624313 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.624288    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jwps4" event={"ID":"827a263b-611a-4335-8db3-1d4a4009fda2","Type":"ContainerStarted","Data":"5a9344df81905e2352e0a1e9567a48532a4030b5cdbbd2fbe8c1342cf829729c"}
Sep 10 23:36:39.625482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.625463    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr968" event={"ID":"ad580eec-bd50-4887-a786-5c9d6971872f","Type":"ContainerStarted","Data":"f51a0ec230deffb0cb71e092807f5fdc45c312dd7bc5142b9ec5650778ae5af6"}
Sep 10 23:36:39.627677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.627656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"089bc3a373d5fdd9bc085012ff247f04d065e8b240d63ed9a8fdd2c28fd09085"}
Sep 10 23:36:39.629741 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.629718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" event={"ID":"6e9761818410ff00337ec1726a7473d4","Type":"ContainerStarted","Data":"e0adc336996219dbe42c14cc65509c640d545a445bc7cd0373313c70a47c0163"}
Sep 10 23:36:39.630807 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.630788    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rmhq" event={"ID":"d567f9a9-22fe-4fc9-82a4-a04294e982e0","Type":"ContainerStarted","Data":"2244dd23f2affca7f0c220ae1218258102ea0981619343ba3b71e58d90dbbee4"}
Sep 10 23:36:39.633738 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.633712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8xph" event={"ID":"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1","Type":"ContainerStarted","Data":"fb62a6bff95e632154abe6da6192870073c1239202aeb5026d95dee458c0d4b7"}
Sep 10 23:36:39.635211 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.635192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerStarted","Data":"77e7d143d751f8efc1b334c49b26e9802940bcb1f11a7d842fecd1cd4a2dfda6"}
Sep 10 23:36:39.642331 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:39.642292    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" podStartSLOduration=2.642281284 podStartE2EDuration="2.642281284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:39.642150989 +0000 UTC m=+3.471978209" watchObservedRunningTime="2026-09-10 23:36:39.642281284 +0000 UTC m=+3.472108503"
Sep 10 23:36:40.138149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.138065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:40.138149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.138133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:40.138362 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.138277    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:40.138362 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.138343    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.13832421 +0000 UTC m=+5.968151429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:40.138809 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.138786    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:40.138916 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.138846    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.138830976 +0000 UTC m=+5.968658190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:40.239185 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.239149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:40.239402 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.239353    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:40.239402 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.239377    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:40.239402 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.239397    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:40.239576 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.239461    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.239442399 +0000 UTC m=+6.069269598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:40.551970 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.551850    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:38 +0000 UTC" deadline="2028-04-21 21:25:39.611568206 +0000 UTC"
Sep 10 23:36:40.551970 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.551897    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14133h48m59.059675789s"
Sep 10 23:36:40.610906 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.610873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:40.611095 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.611026    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:40.611474 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.611454    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:40.611568 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.611552    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:40.611632 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.611615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:40.611708 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:40.611686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:40.646261 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.646226    2574 generic.go:356] "Generic (PLEG): container finished" podID="fabeef8f745298115c8060203460ffaa" containerID="8fd05617e4a2820cc87b81c60e1deef53d0e545eb7f560a97bd7996fd9063e5f" exitCode=0
Sep 10 23:36:40.646441 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:40.646343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"fabeef8f745298115c8060203460ffaa","Type":"ContainerDied","Data":"8fd05617e4a2820cc87b81c60e1deef53d0e545eb7f560a97bd7996fd9063e5f"}
Sep 10 23:36:41.665699 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:41.665255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"fabeef8f745298115c8060203460ffaa","Type":"ContainerStarted","Data":"b559719666967504268a4309cbbba47e79b5d64056da61f4a8d5c413d3eb73df"}
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.155666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.155738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.155795    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.155866    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.155877    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.155856178 +0000 UTC m=+9.985683396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:42.156247 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.155919    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.155902721 +0000 UTC m=+9.985729922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:42.256343 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.256304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:42.256519 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.256486    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:42.256519 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.256506    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:42.256519 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.256519    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:42.256681 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.256577    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:36:46.256560508 +0000 UTC m=+10.086387727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:42.608659 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.608574    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:42.608826 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.608711    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:42.609127 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.609103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:42.609253 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.609206    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:42.609380 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:42.609361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:42.609488 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:42.609468    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:44.609003 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:44.608967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:44.609431 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:44.608967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:44.609431 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:44.609119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:44.609431 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:44.609220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:44.609431 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:44.609219    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:44.609431 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:44.609334    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:46.190450 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.190409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:46.190939 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.190471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:46.190939 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.190622    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:46.190939 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.190685    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.190665853 +0000 UTC m=+18.020493065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:46.191192 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.191084    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:46.191192 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.191161    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.191128763 +0000 UTC m=+18.020955980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:46.291479 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.291371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:46.291639 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.291522    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:46.291639 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.291544    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:46.291639 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.291556    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:46.291639 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.291611    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.291594634 +0000 UTC m=+18.121421834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:46.609167 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.609080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:46.609311 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.609189    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:46.609311 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.609211    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:46.609311 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.609289    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:46.609471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:46.609397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:46.609518 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:46.609478    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:48.608362 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:48.608332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:48.608362 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:48.608353    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:48.608796 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:48.608332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:48.608796 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:48.608455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:48.608796 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:48.608537    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:48.608796 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:48.608609    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:50.608988 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:50.608917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:50.609462 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:50.608917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:50.609462 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:50.609052    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:50.609462 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:50.609069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:50.609462 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:50.609119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:50.609462 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:50.609189    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:52.608345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:52.608305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:52.608345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:52.608341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:52.608888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:52.608305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:52.608888 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:52.608450    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:52.608888 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:52.608544    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:52.608888 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:52.608628    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:54.247777 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.247730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:54.248230 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.247829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:54.248230 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.247875    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:54.248230 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.247947    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:54.248230 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.247959    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:37:10.247938776 +0000 UTC m=+34.077765982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:36:54.248230 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.248000    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:10.247983506 +0000 UTC m=+34.077810709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:36:54.348561 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.348519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:54.348731 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.348673    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:36:54.348731 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.348689    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:36:54.348731 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.348700    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:54.348887 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.348751    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:37:10.348737017 +0000 UTC m=+34.178564214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:36:54.608928 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.608893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:54.608928 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.608930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:54.609152 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:54.608893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:54.609152 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.609018    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:54.609152 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.609115    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:54.609256 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:54.609213    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:56.609206 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:56.609177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:56.609610 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:56.609276    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:56.609610 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:56.609558    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:56.609697 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:56.609648    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:56.609697 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:56.609688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:56.609759 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:56.609733    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:57.699974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.699557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"10f5e2162d7044ff3ba960be57a45688072f15a8916968f4657db663150cbd6c"}
Sep 10 23:36:57.701329 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.701300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rmhq" event={"ID":"d567f9a9-22fe-4fc9-82a4-a04294e982e0","Type":"ContainerStarted","Data":"a75f4bd946744a8d4293c52f92784604354b964ae0943c4ac6aaa6101cd0b315"}
Sep 10 23:36:57.703520 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.703286    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8xph" event={"ID":"83a76ea4-06b0-4a7c-9b1f-aa1300d006f1","Type":"ContainerStarted","Data":"a66b7f066c8c9bdc2bc99a108e66344ae8c0a758361ae2a9654d7671c87ca33e"}
Sep 10 23:36:57.707988 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.707960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerStarted","Data":"498270926fc354617431f1f33dbfcc11c61df99a5aacf93588cb65bf71b068d7"}
Sep 10 23:36:57.711059 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.710973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqz5d" event={"ID":"1c599127-22a5-4cac-9bb0-48fbc8865853","Type":"ContainerStarted","Data":"557f743357c604b0632077ba00ed144c08288b52975a8e818ab8da3b75500e1b"}
Sep 10 23:36:57.712646 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.712609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd" event={"ID":"522b5148-f95b-4ec5-aa9d-59f06fe242e8","Type":"ContainerStarted","Data":"6c7071b39940d0c1a9a88cd4337f2b862e20928093b4314142a79c92e9decd3a"}
Sep 10 23:36:57.713939 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.713894    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4r6t6" event={"ID":"6ab405fb-bc66-4ffa-9116-49881c27576a","Type":"ContainerStarted","Data":"1be3f20dd53c53211102b3c04b85ec465ac91ce132be657573b474cdf4903ee0"}
Sep 10 23:36:57.716394 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.715532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr968" event={"ID":"ad580eec-bd50-4887-a786-5c9d6971872f","Type":"ContainerStarted","Data":"bd9238e91224b1d640d7c0ddaa44765207d6944cf8e02cba2b70b09fbecaf562"}
Sep 10 23:36:57.717933 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.717898    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6rmhq" podStartSLOduration=3.6119982200000003 podStartE2EDuration="21.717888236s" totalImagesPullingTime="18.105890016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.717813958 +0000 UTC m=+21.547641179" watchObservedRunningTime="2026-09-10 23:36:57.717888236 +0000 UTC m=+21.547715455"
Sep 10 23:36:57.718117 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.718096    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" podStartSLOduration=20.718090822 podStartE2EDuration="20.718090822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:41.680117116 +0000 UTC m=+5.509944338" watchObservedRunningTime="2026-09-10 23:36:57.718090822 +0000 UTC m=+21.547918041"
Sep 10 23:36:57.736598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.736545    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z8xph" podStartSLOduration=3.845819126 podStartE2EDuration="21.736525615s" totalImagesPullingTime="17.890706489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.735916924 +0000 UTC m=+21.565744144" watchObservedRunningTime="2026-09-10 23:36:57.736525615 +0000 UTC m=+21.566352835"
Sep 10 23:36:57.767854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.767796    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4r6t6" podStartSLOduration=3.87698164 podStartE2EDuration="21.767780314s" totalImagesPullingTime="17.890798674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.767366488 +0000 UTC m=+21.597193706" watchObservedRunningTime="2026-09-10 23:36:57.767780314 +0000 UTC m=+21.597607549"
Sep 10 23:36:57.802480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.802433    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dr968" podStartSLOduration=3.90241715 podStartE2EDuration="21.802416401s" totalImagesPullingTime="17.899999251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.802164142 +0000 UTC m=+21.631991363" watchObservedRunningTime="2026-09-10 23:36:57.802416401 +0000 UTC m=+21.632243620"
Sep 10 23:36:57.825371 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:57.825326    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mqz5d" podStartSLOduration=7.569094142 podStartE2EDuration="21.825307342s" totalImagesPullingTime="14.2562132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.825159405 +0000 UTC m=+21.654986625" watchObservedRunningTime="2026-09-10 23:36:57.825307342 +0000 UTC m=+21.655134561"
Sep 10 23:36:58.608973 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.608941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:36:58.608973 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.608956    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:36:58.609178 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.608946    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:36:58.609178 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:58.609060    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:36:58.609178 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:58.609132    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:36:58.609283 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:36:58.609197    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:36:58.718645 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.718616    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="498270926fc354617431f1f33dbfcc11c61df99a5aacf93588cb65bf71b068d7" exitCode=0
Sep 10 23:36:58.719413 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.718698    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"498270926fc354617431f1f33dbfcc11c61df99a5aacf93588cb65bf71b068d7"}
Sep 10 23:36:58.722496 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.722469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"a32b04dc943898f80c57ed5e7ac563c31ed5578030ed3c798f40119af664a343"}
Sep 10 23:36:58.722588 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.722568    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"ddd765738ffa3f4b79c742a0ef169ade08b2c781616ad2ac239f126886199752"}
Sep 10 23:36:58.722639 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.722624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"42c3cbc61b8a3a39e4a6adda4947de379bcbe857b626ed08e2b2a606ef515227"}
Sep 10 23:36:58.722692 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.722680    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"4a55de9f512b93dd6bb2102635779aa0a1b4ccd32b509bbef044d1880487b630"}
Sep 10 23:36:58.722740 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.722727    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"b09e981ab73e98622224365746824fca2e58c3cb0dedffda8e671d3f0b97b55a"}
Sep 10 23:36:58.813714 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:58.813479    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:36:59.576816 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.576716    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:36:58.81371289Z","UUID":"d647758e-0f30-4d21-aadf-a6783f4d5bcb","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:36:59.578558 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.578452    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:36:59.578558 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.578491    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:36:59.727686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.727654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd" event={"ID":"522b5148-f95b-4ec5-aa9d-59f06fe242e8","Type":"ContainerStarted","Data":"957447b52ab615e48d447008005d66d5529c0abeb8045ee46a09f04e5ae15796"}
Sep 10 23:36:59.729613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.729587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jwps4" event={"ID":"827a263b-611a-4335-8db3-1d4a4009fda2","Type":"ContainerStarted","Data":"af8a0a19d12583aa22801add58995d7aa7a3aca6f5992b7ec0de902dbdbec7e3"}
Sep 10 23:36:59.743550 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:36:59.743475    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jwps4" podStartSLOduration=5.848887091 podStartE2EDuration="23.743459876s" totalImagesPullingTime="17.894572785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:59.743321021 +0000 UTC m=+23.573148247" watchObservedRunningTime="2026-09-10 23:36:59.743459876 +0000 UTC m=+23.573287094"
Sep 10 23:37:00.608482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.608449    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:00.608744 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.608448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:00.608744 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:00.608576    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:00.608744 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.608606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:00.608744 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:00.608698    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:00.608993 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:00.608805    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:00.734845 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.734807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd" event={"ID":"522b5148-f95b-4ec5-aa9d-59f06fe242e8","Type":"ContainerStarted","Data":"63b09062405f3731f9c9af863dd1aaecf2ab780dd0bb0e640eaf2f10d4b52bd6"}
Sep 10 23:37:00.738968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.738932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"a30dd685a606fd30f794d5c9188f32326cb9232d8e26c04b426f3172bf6f4513"}
Sep 10 23:37:00.753583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:00.753532    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g9xhd" podStartSLOduration=4.835987054 podStartE2EDuration="24.753514006s" totalImagesPullingTime="19.917526952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:00.752707667 +0000 UTC m=+24.582534911" watchObservedRunningTime="2026-09-10 23:37:00.753514006 +0000 UTC m=+24.583341228"
Sep 10 23:37:02.227602 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.227569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:37:02.228353 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.228333    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:37:02.608284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.608256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:02.608284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.608271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:02.608441 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.608263    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:02.608441 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:02.608356    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:02.608441 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:02.608434    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:02.608611 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:02.608511    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:02.750099 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.749958    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" event={"ID":"fc2bcda4-346d-4bc5-9b7c-c37d9ee029f9","Type":"ContainerStarted","Data":"ada60ed9687cb285d1402b5755a00fcd34ebeab4cbc7e39b19b6addefd565648"}
Sep 10 23:37:02.750244 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.750163    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:37:02.750599 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.750585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mqz5d"
Sep 10 23:37:02.774881 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:02.774836    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm" podStartSLOduration=8.839518234 podStartE2EDuration="26.774822357s" totalImagesPullingTime="17.935304123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:02.774643723 +0000 UTC m=+26.604470950" watchObservedRunningTime="2026-09-10 23:37:02.774822357 +0000 UTC m=+26.604649576"
Sep 10 23:37:03.753822 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.753793    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="5a55e43de0cb0421c03f773bf0a3e8ef9304135d3e4099339e39dc31dcc1cb16" exitCode=0
Sep 10 23:37:03.754417 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.753868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"5a55e43de0cb0421c03f773bf0a3e8ef9304135d3e4099339e39dc31dcc1cb16"}
Sep 10 23:37:03.754417 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.754366    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:03.754532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.754439    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:03.754532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.754489    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:03.769091 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.769071    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:03.769311 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:03.769299    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:04.545042 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.543680    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-528jm"]
Sep 10 23:37:04.545042 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.544168    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:04.545042 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:04.544315    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:04.547093 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.546962    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h28ql"]
Sep 10 23:37:04.547238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.547195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:04.547304 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:04.547276    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:04.547748 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.547700    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-msf67"]
Sep 10 23:37:04.547862 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:04.547819    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:04.547935 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:04.547913    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:05.608310 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:05.608286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:05.608832 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:05.608390    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:05.759806 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:05.759760    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="93c4c36e3823e2f36787cfeb013c9c54d614dc2c30e972f6506d1527d0ef2ddb" exitCode=0
Sep 10 23:37:05.759922 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:05.759849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"93c4c36e3823e2f36787cfeb013c9c54d614dc2c30e972f6506d1527d0ef2ddb"}
Sep 10 23:37:06.609648 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:06.609615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:06.610075 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:06.609721    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:06.610075 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:06.609807    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:06.610075 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:06.609895    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:07.608692 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:07.608650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:07.608895 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:07.608812    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:07.767469 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:07.767430    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="0f4d2f21ae4a5b2eeebf4fea0e6b437437e1a3d7e032696ef9a1e5087f8046fd" exitCode=0
Sep 10 23:37:07.767848 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:07.767486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"0f4d2f21ae4a5b2eeebf4fea0e6b437437e1a3d7e032696ef9a1e5087f8046fd"}
Sep 10 23:37:08.608752 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:08.608723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:08.608963 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:08.608862    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:08.608963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:08.608897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:08.609088 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:08.609004    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:09.608753 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:09.608720    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:09.609305 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:09.608863    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-msf67" podUID="906f16ef-b837-4aac-b339-f93860a0053a"
Sep 10 23:37:10.266289 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:10.266252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:10.266480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:10.266328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:10.266480 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.266419    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:37:10.266480 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.266431    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:37:10.266615 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.266497    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs podName:906f16ef-b837-4aac-b339-f93860a0053a nodeName:}" failed. No retries permitted until 2026-09-10 23:37:42.266479432 +0000 UTC m=+66.096306634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs") pod "network-metrics-daemon-msf67" (UID: "906f16ef-b837-4aac-b339-f93860a0053a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:37:10.266615 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.266520    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret podName:53618be8-c85c-4f61-acb4-1e738be631f0 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:42.266509831 +0000 UTC m=+66.096337030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret") pod "global-pull-secret-syncer-528jm" (UID: "53618be8-c85c-4f61-acb4-1e738be631f0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:37:10.366893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:10.366864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:10.367079 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.366999    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:37:10.367079 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.367017    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:37:10.367079 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.367030    2574 projected.go:196] Error preparing data for projected volume kube-api-access-w68dz for pod openshift-network-diagnostics/network-check-target-h28ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:37:10.367247 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.367090    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz podName:6bc07400-5b6d-4751-8c45-2b5db79f91dc nodeName:}" failed. No retries permitted until 2026-09-10 23:37:42.367073541 +0000 UTC m=+66.196900751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w68dz" (UniqueName: "kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz") pod "network-check-target-h28ql" (UID: "6bc07400-5b6d-4751-8c45-2b5db79f91dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:37:10.608820 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:10.608787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:10.609229 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.608908    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-528jm" podUID="53618be8-c85c-4f61-acb4-1e738be631f0"
Sep 10 23:37:10.609229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:10.608936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:10.609229 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:10.609053    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h28ql" podUID="6bc07400-5b6d-4751-8c45-2b5db79f91dc"
Sep 10 23:37:11.456562 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.456527    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeReady"
Sep 10 23:37:11.456719 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.456663    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:37:11.496086 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.496055    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:37:11.498561 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.498533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.502954 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.502927    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 23:37:11.503132 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.503103    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 23:37:11.503132 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.503129    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wsf9n\""
Sep 10 23:37:11.503308 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.503290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 23:37:11.510438 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.510419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 23:37:11.518357 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.518340    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9jhd"]
Sep 10 23:37:11.521008 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.520988    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.522847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.522792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:37:11.523530 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.523514    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9q52c"]
Sep 10 23:37:11.525275 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.525256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.528404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.528206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:37:11.528404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.528288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9sdk\""
Sep 10 23:37:11.528568 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.528459    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:37:11.532480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.532463    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:37:11.532578 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.532503    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:37:11.532797 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.532778    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qfssj\""
Sep 10 23:37:11.534534 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.534514    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:37:11.542866 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.542844    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9jhd"]
Sep 10 23:37:11.545147 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.545119    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9q52c"]
Sep 10 23:37:11.576062 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44qs\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576157 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576157 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576102    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576157 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576157 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576290 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576290 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.576290 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.576237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.608086 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.608062    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:11.611672 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.611652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:37:11.611974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.611675    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\""
Sep 10 23:37:11.677399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fa81825-3249-49be-8c46-e61dce20883a-cert\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.677511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vftm\" (UniqueName: \"kubernetes.io/projected/9fa81825-3249-49be-8c46-e61dce20883a-kube-api-access-9vftm\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.677511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de00e982-77f5-4483-84a1-54e3d5c778ba-config-volume\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdb4\" (UniqueName: \"kubernetes.io/projected/de00e982-77f5-4483-84a1-54e3d5c778ba-kube-api-access-nxdb4\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:11.677556    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44qs\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:11.677575    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5f446cc6-f9gb9: secret "image-registry-tls" not found
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:11.677633    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls podName:cc2a8555-25a2-491d-bcd1-8a8794de15c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.177614702 +0000 UTC m=+36.007441899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls") pod "image-registry-f5f446cc6-f9gb9" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3") : secret "image-registry-tls" not found
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.677794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de00e982-77f5-4483-84a1-54e3d5c778ba-tmp-dir\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.678284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.677808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.678284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.678147    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.678381 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.678365    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.678433 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.678415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.681135 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.681117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.681225 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.681198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.696309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.696288    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44qs\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.697561 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.697533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:11.778202 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fa81825-3249-49be-8c46-e61dce20883a-cert\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.778314 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vftm\" (UniqueName: \"kubernetes.io/projected/9fa81825-3249-49be-8c46-e61dce20883a-kube-api-access-9vftm\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.778386 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de00e982-77f5-4483-84a1-54e3d5c778ba-config-volume\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.778442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdb4\" (UniqueName: \"kubernetes.io/projected/de00e982-77f5-4483-84a1-54e3d5c778ba-kube-api-access-nxdb4\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.778604 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.778676 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de00e982-77f5-4483-84a1-54e3d5c778ba-tmp-dir\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.778728 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:11.778708    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:11.778845 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:11.778823    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls podName:de00e982-77f5-4483-84a1-54e3d5c778ba nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.278792319 +0000 UTC m=+36.108619518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls") pod "dns-default-r9jhd" (UID: "de00e982-77f5-4483-84a1-54e3d5c778ba") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:11.778966 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.778950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de00e982-77f5-4483-84a1-54e3d5c778ba-tmp-dir\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.779119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.779091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de00e982-77f5-4483-84a1-54e3d5c778ba-config-volume\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.780836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.780817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fa81825-3249-49be-8c46-e61dce20883a-cert\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.783039 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.783013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerStarted","Data":"5a5d486fcb62e104ffc0d03e60a680107be7c5dc961793c0931b0615bbbb5e84"}
Sep 10 23:37:11.789669 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.789650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdb4\" (UniqueName: \"kubernetes.io/projected/de00e982-77f5-4483-84a1-54e3d5c778ba-kube-api-access-nxdb4\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:11.790141 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.790121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vftm\" (UniqueName: \"kubernetes.io/projected/9fa81825-3249-49be-8c46-e61dce20883a-kube-api-access-9vftm\") pod \"ingress-canary-9q52c\" (UID: \"9fa81825-3249-49be-8c46-e61dce20883a\") " pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:11.840199 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:11.840174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9q52c"
Sep 10 23:37:12.112139 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.112112    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9q52c"]
Sep 10 23:37:12.115846 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:12.115814    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa81825_3249_49be_8c46_e61dce20883a.slice/crio-aca2df219fdc691bf09e8029e5f2a814d5e5e3c1b173c462deff722ba1e36ce2 WatchSource:0}: Error finding container aca2df219fdc691bf09e8029e5f2a814d5e5e3c1b173c462deff722ba1e36ce2: Status 404 returned error can't find the container with id aca2df219fdc691bf09e8029e5f2a814d5e5e3c1b173c462deff722ba1e36ce2
Sep 10 23:37:12.181238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.181215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:12.181368 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:12.181353    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:12.181412 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:12.181370    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5f446cc6-f9gb9: secret "image-registry-tls" not found
Sep 10 23:37:12.181455 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:12.181446    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls podName:cc2a8555-25a2-491d-bcd1-8a8794de15c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:13.181432305 +0000 UTC m=+37.011259507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls") pod "image-registry-f5f446cc6-f9gb9" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3") : secret "image-registry-tls" not found
Sep 10 23:37:12.282353 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.282324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:12.282501 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:12.282482    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:12.282554 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:12.282545    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls podName:de00e982-77f5-4483-84a1-54e3d5c778ba nodeName:}" failed. No retries permitted until 2026-09-10 23:37:13.282529376 +0000 UTC m=+37.112356578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls") pod "dns-default-r9jhd" (UID: "de00e982-77f5-4483-84a1-54e3d5c778ba") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:12.608326 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.608299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:12.608482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.608299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:12.614352 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.614320    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:37:12.615008 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.614420    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\""
Sep 10 23:37:12.615008 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.614476    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:37:12.615008 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.614422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:37:12.786824 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.786776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9q52c" event={"ID":"9fa81825-3249-49be-8c46-e61dce20883a","Type":"ContainerStarted","Data":"aca2df219fdc691bf09e8029e5f2a814d5e5e3c1b173c462deff722ba1e36ce2"}
Sep 10 23:37:12.790258 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.790229    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="5a5d486fcb62e104ffc0d03e60a680107be7c5dc961793c0931b0615bbbb5e84" exitCode=0
Sep 10 23:37:12.790393 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:12.790270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"5a5d486fcb62e104ffc0d03e60a680107be7c5dc961793c0931b0615bbbb5e84"}
Sep 10 23:37:13.189311 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.189227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:13.189438 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:13.189382    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:13.189438 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:13.189403    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5f446cc6-f9gb9: secret "image-registry-tls" not found
Sep 10 23:37:13.189540 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:13.189466    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls podName:cc2a8555-25a2-491d-bcd1-8a8794de15c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.189444784 +0000 UTC m=+39.019271982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls") pod "image-registry-f5f446cc6-f9gb9" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3") : secret "image-registry-tls" not found
Sep 10 23:37:13.289810 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.289753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:13.289983 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:13.289886    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:13.289983 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:13.289957    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls podName:de00e982-77f5-4483-84a1-54e3d5c778ba nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.289935343 +0000 UTC m=+39.119762550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls") pod "dns-default-r9jhd" (UID: "de00e982-77f5-4483-84a1-54e3d5c778ba") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:13.796517 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.796261    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae77854-f07b-45ff-8684-69dc0ae9d1b2" containerID="94f64b1258dc21aa9436976e77ffdca164ce9e6a92cf6effcd209f83ea753c04" exitCode=0
Sep 10 23:37:13.796985 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.796351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerDied","Data":"94f64b1258dc21aa9436976e77ffdca164ce9e6a92cf6effcd209f83ea753c04"}
Sep 10 23:37:13.908902 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.908869    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zgzbk"]
Sep 10 23:37:13.911488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.911469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:13.913816 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.913795    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bptlm\""
Sep 10 23:37:13.915328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.914978    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:37:13.915328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.915065    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:37:13.915328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.915104    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:37:13.915328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.915269    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:37:13.923959 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.923937    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zgzbk"]
Sep 10 23:37:13.995025 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.994993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnc66\" (UniqueName: \"kubernetes.io/projected/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-api-access-cnc66\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:13.995165 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.995071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-crio-socket\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:13.995229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.995162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-data-volume\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:13.995229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.995198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:13.995314 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:13.995227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096277 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-crio-socket\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096430 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-data-volume\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096430 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096430 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-crio-socket\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096581 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.096544    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.096677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnc66\" (UniqueName: \"kubernetes.io/projected/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-api-access-cnc66\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096677 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.096624    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.596603356 +0000 UTC m=+38.426430575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.096814 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.096681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-data-volume\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.096814 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.096691    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.096814 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.096744    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.596727348 +0000 UTC m=+38.426554548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.100588 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.100569    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"]
Sep 10 23:37:14.102435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.102416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.104340 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.104318    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 23:37:14.105077 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.105060    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5x2k7\""
Sep 10 23:37:14.105182 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.105089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 23:37:14.109324 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.109248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnc66\" (UniqueName: \"kubernetes.io/projected/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-api-access-cnc66\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.112482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.112461    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"]
Sep 10 23:37:14.197253 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.197190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1017b2c6-48eb-4ef0-8a08-642d2402ee55-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.197407 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.197274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.297833 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.297781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1017b2c6-48eb-4ef0-8a08-642d2402ee55-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.298017 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.297860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.298089 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.298031    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 23:37:14.298137 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.298112    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert podName:1017b2c6-48eb-4ef0-8a08-642d2402ee55 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.798090131 +0000 UTC m=+38.627917332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-76jnj" (UID: "1017b2c6-48eb-4ef0-8a08-642d2402ee55") : secret "networking-console-plugin-cert" not found
Sep 10 23:37:14.298585 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.298552    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1017b2c6-48eb-4ef0-8a08-642d2402ee55-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.601542 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.601471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.601542 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.601510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:14.601741 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.601637    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.601741 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.601698    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.601741 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.601729    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.601703277 +0000 UTC m=+39.431530494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:14.601872 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.601754    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.601743583 +0000 UTC m=+39.431570782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:14.800618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.800580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9q52c" event={"ID":"9fa81825-3249-49be-8c46-e61dce20883a","Type":"ContainerStarted","Data":"20fc6fefadbeef8b4d99a6d8a108181d4dea2cb2d5989d8db4fc5481096589e9"}
Sep 10 23:37:14.803038 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.803016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:14.803218 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.803201    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 23:37:14.803279 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:14.803262    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert podName:1017b2c6-48eb-4ef0-8a08-642d2402ee55 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.803243052 +0000 UTC m=+39.633070248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-76jnj" (UID: "1017b2c6-48eb-4ef0-8a08-642d2402ee55") : secret "networking-console-plugin-cert" not found
Sep 10 23:37:14.804187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.804166    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frwgx" event={"ID":"5ae77854-f07b-45ff-8684-69dc0ae9d1b2","Type":"ContainerStarted","Data":"6226fc3ae29e6f144531992cb1df7c613679725ee29e714f075b19959a75dedd"}
Sep 10 23:37:14.816453 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.816410    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9q52c" podStartSLOduration=1.586050068 podStartE2EDuration="3.816395533s" totalImagesPullingTime="2.230345465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:14.815921411 +0000 UTC m=+38.645748643" watchObservedRunningTime="2026-09-10 23:37:14.816395533 +0000 UTC m=+38.646222753"
Sep 10 23:37:14.840172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:14.840126    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-frwgx" podStartSLOduration=11.073320449 podStartE2EDuration="38.840109028s" totalImagesPullingTime="27.766788579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:14.839103892 +0000 UTC m=+38.668931113" watchObservedRunningTime="2026-09-10 23:37:14.840109028 +0000 UTC m=+38.669936248"
Sep 10 23:37:15.207811 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.207753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:15.207957 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.207930    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:15.207957 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.207947    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5f446cc6-f9gb9: secret "image-registry-tls" not found
Sep 10 23:37:15.208041 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.208006    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls podName:cc2a8555-25a2-491d-bcd1-8a8794de15c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.207987718 +0000 UTC m=+43.037815162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls") pod "image-registry-f5f446cc6-f9gb9" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3") : secret "image-registry-tls" not found
Sep 10 23:37:15.308323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.308297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:15.308452 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.308414    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:15.308490 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.308459    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls podName:de00e982-77f5-4483-84a1-54e3d5c778ba nodeName:}" failed. No retries permitted until 2026-09-10 23:37:19.308446672 +0000 UTC m=+43.138273869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls") pod "dns-default-r9jhd" (UID: "de00e982-77f5-4483-84a1-54e3d5c778ba") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:15.319530 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.319499    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xldvn"]
Sep 10 23:37:15.343978 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.343959    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"]
Sep 10 23:37:15.344114 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.344094    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.347439 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.347414    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 23:37:15.347659 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.347646    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 23:37:15.347713 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.347650    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 23:37:15.347852 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.347837    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vnzxz\""
Sep 10 23:37:15.347972 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.347955    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 23:37:15.367745 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.367725    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"]
Sep 10 23:37:15.367846 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.367784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xldvn"]
Sep 10 23:37:15.367889 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.367850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"
Sep 10 23:37:15.370240 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.370224    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 23:37:15.370323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.370224    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-z9pbg\""
Sep 10 23:37:15.370323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.370273    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 23:37:15.510225 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.510158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4gs\" (UniqueName: \"kubernetes.io/projected/855738bf-3722-4a10-9f03-e5f4b9062b1d-kube-api-access-tr4gs\") pod \"migrator-8b4d697c5-c5h8l\" (UID: \"855738bf-3722-4a10-9f03-e5f4b9062b1d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"
Sep 10 23:37:15.510225 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.510193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-key\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.510225 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.510219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-cabundle\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.510418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.510322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskb2\" (UniqueName: \"kubernetes.io/projected/cca499bb-2848-4a6a-98f8-56eaa305a2c1-kube-api-access-hskb2\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.610988 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.610958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskb2\" (UniqueName: \"kubernetes.io/projected/cca499bb-2848-4a6a-98f8-56eaa305a2c1-kube-api-access-hskb2\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.611133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:15.611133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:15.611133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4gs\" (UniqueName: \"kubernetes.io/projected/855738bf-3722-4a10-9f03-e5f4b9062b1d-kube-api-access-tr4gs\") pod \"migrator-8b4d697c5-c5h8l\" (UID: \"855738bf-3722-4a10-9f03-e5f4b9062b1d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"
Sep 10 23:37:15.611133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-key\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.611133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-cabundle\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.611346 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.611194    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:15.611346 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.611193    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:15.611346 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.611252    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:17.611238274 +0000 UTC m=+41.441065489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:15.611346 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.611299    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:17.611279912 +0000 UTC m=+41.441107126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:15.611850 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.611830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-cabundle\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.614493 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.614474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cca499bb-2848-4a6a-98f8-56eaa305a2c1-signing-key\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.618836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.618812    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskb2\" (UniqueName: \"kubernetes.io/projected/cca499bb-2848-4a6a-98f8-56eaa305a2c1-kube-api-access-hskb2\") pod \"service-ca-568869fdc5-xldvn\" (UID: \"cca499bb-2848-4a6a-98f8-56eaa305a2c1\") " pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.618927 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.618843    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4gs\" (UniqueName: \"kubernetes.io/projected/855738bf-3722-4a10-9f03-e5f4b9062b1d-kube-api-access-tr4gs\") pod \"migrator-8b4d697c5-c5h8l\" (UID: \"855738bf-3722-4a10-9f03-e5f4b9062b1d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"
Sep 10 23:37:15.652731 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.652711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xldvn"
Sep 10 23:37:15.668201 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.668183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8xph_83a76ea4-06b0-4a7c-9b1f-aa1300d006f1/dns-node-resolver/0.log"
Sep 10 23:37:15.675336 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.675321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"
Sep 10 23:37:15.812810 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.812734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:15.813096 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.812850    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 23:37:15.813096 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:15.812905    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert podName:1017b2c6-48eb-4ef0-8a08-642d2402ee55 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:17.812891782 +0000 UTC m=+41.642718979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-76jnj" (UID: "1017b2c6-48eb-4ef0-8a08-642d2402ee55") : secret "networking-console-plugin-cert" not found
Sep 10 23:37:15.946878 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.946846    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xldvn"]
Sep 10 23:37:15.950503 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:15.950476    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca499bb_2848_4a6a_98f8_56eaa305a2c1.slice/crio-e60d0a796ab8ab7104002f10d5249c4a18175684013418e97b5e488882a9e432 WatchSource:0}: Error finding container e60d0a796ab8ab7104002f10d5249c4a18175684013418e97b5e488882a9e432: Status 404 returned error can't find the container with id e60d0a796ab8ab7104002f10d5249c4a18175684013418e97b5e488882a9e432
Sep 10 23:37:15.972611 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:15.972587    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l"]
Sep 10 23:37:15.975648 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:15.975623    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855738bf_3722_4a10_9f03_e5f4b9062b1d.slice/crio-707b43588c1edb5a269ceb9c8e7fb0ec62ac51d8e0558fcc3047f6f4d533c5c0 WatchSource:0}: Error finding container 707b43588c1edb5a269ceb9c8e7fb0ec62ac51d8e0558fcc3047f6f4d533c5c0: Status 404 returned error can't find the container with id 707b43588c1edb5a269ceb9c8e7fb0ec62ac51d8e0558fcc3047f6f4d533c5c0
Sep 10 23:37:16.467273 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:16.467245    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4r6t6_6ab405fb-bc66-4ffa-9116-49881c27576a/node-ca/0.log"
Sep 10 23:37:16.811415 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:16.811327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l" event={"ID":"855738bf-3722-4a10-9f03-e5f4b9062b1d","Type":"ContainerStarted","Data":"707b43588c1edb5a269ceb9c8e7fb0ec62ac51d8e0558fcc3047f6f4d533c5c0"}
Sep 10 23:37:16.812705 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:16.812677    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xldvn" event={"ID":"cca499bb-2848-4a6a-98f8-56eaa305a2c1","Type":"ContainerStarted","Data":"e60d0a796ab8ab7104002f10d5249c4a18175684013418e97b5e488882a9e432"}
Sep 10 23:37:17.067846 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:17.067756    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9q52c_9fa81825-3249-49be-8c46-e61dce20883a/serve-healthcheck-canary/0.log"
Sep 10 23:37:17.635916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:17.635872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:17.636111 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:17.635935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:17.636111 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.636056    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 23:37:17.636111 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.636084    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:17.636266 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.636148    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:21.636125998 +0000 UTC m=+45.465953218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : configmap "kube-rbac-proxy" not found
Sep 10 23:37:17.636266 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.636173    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:21.636163055 +0000 UTC m=+45.465990255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:17.837480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:17.837441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:17.837657 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.837571    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 23:37:17.837657 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:17.837655    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert podName:1017b2c6-48eb-4ef0-8a08-642d2402ee55 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:21.837623519 +0000 UTC m=+45.667450723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-76jnj" (UID: "1017b2c6-48eb-4ef0-8a08-642d2402ee55") : secret "networking-console-plugin-cert" not found
Sep 10 23:37:18.819197 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:18.819119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l" event={"ID":"855738bf-3722-4a10-9f03-e5f4b9062b1d","Type":"ContainerStarted","Data":"4b7dc70bdb2711af9436bfc8fcb560c4ba31263eefcc52ab4ff6de6a03303a50"}
Sep 10 23:37:18.819197 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:18.819177    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l" event={"ID":"855738bf-3722-4a10-9f03-e5f4b9062b1d","Type":"ContainerStarted","Data":"dab083960297b2768ba84f39dd9c7b58b754ab0682d04e7ce1a6a171579286ec"}
Sep 10 23:37:18.820682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:18.820659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xldvn" event={"ID":"cca499bb-2848-4a6a-98f8-56eaa305a2c1","Type":"ContainerStarted","Data":"b2bfc44cf17ca34c8dd260f1a785dc47635e603a50f54599e0efe6d94791b627"}
Sep 10 23:37:18.857807 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:18.857742    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-c5h8l" podStartSLOduration=1.76726963 podStartE2EDuration="3.857728108s" totalImagesPullingTime="2.090458478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:18.856350401 +0000 UTC m=+42.686177621" watchObservedRunningTime="2026-09-10 23:37:18.857728108 +0000 UTC m=+42.687555328"
Sep 10 23:37:18.879165 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:18.879120    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-xldvn" podStartSLOduration=1.384124398 podStartE2EDuration="3.879098674s" totalImagesPullingTime="2.494974276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:18.878881915 +0000 UTC m=+42.708709144" watchObservedRunningTime="2026-09-10 23:37:18.879098674 +0000 UTC m=+42.708925894"
Sep 10 23:37:19.248247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:19.248202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:19.248511 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:19.248492    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:37:19.248596 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:19.248514    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5f446cc6-f9gb9: secret "image-registry-tls" not found
Sep 10 23:37:19.248596 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:19.248574    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls podName:cc2a8555-25a2-491d-bcd1-8a8794de15c3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:27.248556007 +0000 UTC m=+51.078383218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls") pod "image-registry-f5f446cc6-f9gb9" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3") : secret "image-registry-tls" not found
Sep 10 23:37:19.349560 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:19.349523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:19.349710 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:19.349662    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:37:19.349760 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:19.349726    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls podName:de00e982-77f5-4483-84a1-54e3d5c778ba nodeName:}" failed. No retries permitted until 2026-09-10 23:37:27.349707617 +0000 UTC m=+51.179534833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls") pod "dns-default-r9jhd" (UID: "de00e982-77f5-4483-84a1-54e3d5c778ba") : secret "dns-default-metrics-tls" not found
Sep 10 23:37:21.667936 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:21.667905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:21.668443 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:21.667943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:21.668443 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:21.668049    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:21.668443 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:21.668101    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls podName:6634f51d-a9bc-43aa-bbe0-dbf3837770ca nodeName:}" failed. No retries permitted until 2026-09-10 23:37:29.668087449 +0000 UTC m=+53.497914646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zgzbk" (UID: "6634f51d-a9bc-43aa-bbe0-dbf3837770ca") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:37:21.668443 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:21.668428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:21.870485 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:21.870450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:21.870657 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:21.870590    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 23:37:21.870721 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:37:21.870672    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert podName:1017b2c6-48eb-4ef0-8a08-642d2402ee55 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:29.870650824 +0000 UTC m=+53.700478035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-76jnj" (UID: "1017b2c6-48eb-4ef0-8a08-642d2402ee55") : secret "networking-console-plugin-cert" not found
Sep 10 23:37:27.317103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.317057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:27.319648 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.319616    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"image-registry-f5f446cc6-f9gb9\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") " pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:27.409814 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.409752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:27.418018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.417990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:27.420251 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.420232    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de00e982-77f5-4483-84a1-54e3d5c778ba-metrics-tls\") pod \"dns-default-r9jhd\" (UID: \"de00e982-77f5-4483-84a1-54e3d5c778ba\") " pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:27.434127 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.434103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:27.698385 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.698350    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:37:27.701445 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:27.701416    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a8555_25a2_491d_bcd1_8a8794de15c3.slice/crio-ea2b06d30fcbd937c2ed7fd27f5602b1a084d374a99df6d8d245f606ddfbf055 WatchSource:0}: Error finding container ea2b06d30fcbd937c2ed7fd27f5602b1a084d374a99df6d8d245f606ddfbf055: Status 404 returned error can't find the container with id ea2b06d30fcbd937c2ed7fd27f5602b1a084d374a99df6d8d245f606ddfbf055
Sep 10 23:37:27.722822 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.722794    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9jhd"]
Sep 10 23:37:27.725689 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:27.725545    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde00e982_77f5_4483_84a1_54e3d5c778ba.slice/crio-fbf858337248dabb05b3b095f29a17c383d54a17955a2178667ee2f51c28f524 WatchSource:0}: Error finding container fbf858337248dabb05b3b095f29a17c383d54a17955a2178667ee2f51c28f524: Status 404 returned error can't find the container with id fbf858337248dabb05b3b095f29a17c383d54a17955a2178667ee2f51c28f524
Sep 10 23:37:27.844917 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.844878    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9jhd" event={"ID":"de00e982-77f5-4483-84a1-54e3d5c778ba","Type":"ContainerStarted","Data":"fbf858337248dabb05b3b095f29a17c383d54a17955a2178667ee2f51c28f524"}
Sep 10 23:37:27.846274 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.846253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" event={"ID":"cc2a8555-25a2-491d-bcd1-8a8794de15c3","Type":"ContainerStarted","Data":"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"}
Sep 10 23:37:27.846376 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.846289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" event={"ID":"cc2a8555-25a2-491d-bcd1-8a8794de15c3","Type":"ContainerStarted","Data":"ea2b06d30fcbd937c2ed7fd27f5602b1a084d374a99df6d8d245f606ddfbf055"}
Sep 10 23:37:27.846427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.846381    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:27.864300 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:27.864242    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" podStartSLOduration=37.86422653 podStartE2EDuration="37.86422653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:27.863825477 +0000 UTC m=+51.693652697" watchObservedRunningTime="2026-09-10 23:37:27.86422653 +0000 UTC m=+51.694053748"
Sep 10 23:37:29.740287 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.740242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:29.742633 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.742607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6634f51d-a9bc-43aa-bbe0-dbf3837770ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zgzbk\" (UID: \"6634f51d-a9bc-43aa-bbe0-dbf3837770ca\") " pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:29.823312 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.823265    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zgzbk"
Sep 10 23:37:29.854301 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.854266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9jhd" event={"ID":"de00e982-77f5-4483-84a1-54e3d5c778ba","Type":"ContainerStarted","Data":"10dc924c320db6bc8d6ce9029a911cd58a46e080c2db4b49c20a6aab5fb1924e"}
Sep 10 23:37:29.854479 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.854319    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9jhd" event={"ID":"de00e982-77f5-4483-84a1-54e3d5c778ba","Type":"ContainerStarted","Data":"c2e9042499ca6909e649dc688c6f79312c15082b2dd121d4d859ec0748522cf2"}
Sep 10 23:37:29.854479 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.854438    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:29.873517 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.873409    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9jhd" podStartSLOduration=17.294067176 podStartE2EDuration="18.873384444s" totalImagesPullingTime="1.579317268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:29.872542786 +0000 UTC m=+53.702370007" watchObservedRunningTime="2026-09-10 23:37:29.873384444 +0000 UTC m=+53.703211666"
Sep 10 23:37:29.942177 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.942133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:29.944520 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:29.944498    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1017b2c6-48eb-4ef0-8a08-642d2402ee55-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-76jnj\" (UID: \"1017b2c6-48eb-4ef0-8a08-642d2402ee55\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:30.016829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.016724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"
Sep 10 23:37:30.095354 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.095322    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zgzbk"]
Sep 10 23:37:30.098389 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:30.098363    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6634f51d_a9bc_43aa_bbe0_dbf3837770ca.slice/crio-723bee37284645f1b5c0096398c153496691e0cbf548e949abf10a031f8b5416 WatchSource:0}: Error finding container 723bee37284645f1b5c0096398c153496691e0cbf548e949abf10a031f8b5416: Status 404 returned error can't find the container with id 723bee37284645f1b5c0096398c153496691e0cbf548e949abf10a031f8b5416
Sep 10 23:37:30.285035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.284956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj"]
Sep 10 23:37:30.287960 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:30.287931    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1017b2c6_48eb_4ef0_8a08_642d2402ee55.slice/crio-0a2aba1ecc8d1aa0bb2cf883f7e757937762d7a22b58fcac6bf0989f1c08b997 WatchSource:0}: Error finding container 0a2aba1ecc8d1aa0bb2cf883f7e757937762d7a22b58fcac6bf0989f1c08b997: Status 404 returned error can't find the container with id 0a2aba1ecc8d1aa0bb2cf883f7e757937762d7a22b58fcac6bf0989f1c08b997
Sep 10 23:37:30.862865 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.862758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj" event={"ID":"1017b2c6-48eb-4ef0-8a08-642d2402ee55","Type":"ContainerStarted","Data":"0a2aba1ecc8d1aa0bb2cf883f7e757937762d7a22b58fcac6bf0989f1c08b997"}
Sep 10 23:37:30.864584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.864553    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zgzbk" event={"ID":"6634f51d-a9bc-43aa-bbe0-dbf3837770ca","Type":"ContainerStarted","Data":"1d7113394038ca4e84a7d3c3bb0e3ebf3791966ec345bda2ce04b04489a455d5"}
Sep 10 23:37:30.864686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.864613    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zgzbk" event={"ID":"6634f51d-a9bc-43aa-bbe0-dbf3837770ca","Type":"ContainerStarted","Data":"56ee25c21c97e65258cbe79ae1fd452b1377d2a14af4a18824794263d1879407"}
Sep 10 23:37:30.864686 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:30.864641    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zgzbk" event={"ID":"6634f51d-a9bc-43aa-bbe0-dbf3837770ca","Type":"ContainerStarted","Data":"723bee37284645f1b5c0096398c153496691e0cbf548e949abf10a031f8b5416"}
Sep 10 23:37:32.872394 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:32.872355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj" event={"ID":"1017b2c6-48eb-4ef0-8a08-642d2402ee55","Type":"ContainerStarted","Data":"93748a88aaf5e68d7dfdfc2098144d201362b80cc8713ea9e1d40d26bfb5765b"}
Sep 10 23:37:32.874389 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:32.874361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zgzbk" event={"ID":"6634f51d-a9bc-43aa-bbe0-dbf3837770ca","Type":"ContainerStarted","Data":"810bf38503757a51521e814e69e9c4b8a6d91ff5cc36f410c43526211717ee25"}
Sep 10 23:37:32.890509 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:32.890469    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-76jnj" podStartSLOduration=17.090040138 podStartE2EDuration="18.890456763s" totalImagesPullingTime="1.800416625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:32.8892939 +0000 UTC m=+56.719121119" watchObservedRunningTime="2026-09-10 23:37:32.890456763 +0000 UTC m=+56.720283982"
Sep 10 23:37:32.908682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:32.908642    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zgzbk" podStartSLOduration=18.025114599 podStartE2EDuration="19.908623364s" totalImagesPullingTime="1.883508765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:32.908351789 +0000 UTC m=+56.738179009" watchObservedRunningTime="2026-09-10 23:37:32.908623364 +0000 UTC m=+56.738450584"
Sep 10 23:37:35.336290 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.336261    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-zfs8p"]
Sep 10 23:37:35.366547 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.366517    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zfs8p"]
Sep 10 23:37:35.366704 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.366614    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:35.367190 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.367171    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:37:35.371815 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.371795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:37:35.371815 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.371804    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hc2tr\""
Sep 10 23:37:35.374473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.374447    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:37:35.444187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.444161    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"]
Sep 10 23:37:35.460061 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.460039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.464373 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.464352    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"]
Sep 10 23:37:35.478473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.478447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcvk\" (UniqueName: \"kubernetes.io/projected/57f24d2a-f57a-4a1a-93a5-180bafac42fb-kube-api-access-spcvk\") pod \"downloads-76f99584f4-zfs8p\" (UID: \"57f24d2a-f57a-4a1a-93a5-180bafac42fb\") " pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:35.579163 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-bound-sa-token\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579267 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcvk\" (UniqueName: \"kubernetes.io/projected/57f24d2a-f57a-4a1a-93a5-180bafac42fb-kube-api-access-spcvk\") pod \"downloads-76f99584f4-zfs8p\" (UID: \"57f24d2a-f57a-4a1a-93a5-180bafac42fb\") " pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:35.579267 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-trusted-ca\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579267 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndgl\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-kube-api-access-bndgl\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-installation-pull-secrets\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6c46-0298-4d37-bab0-56acfc42d99a-ca-trust-extracted\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579451 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-tls\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579484 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-image-registry-private-configuration\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.579484 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.579478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-certificates\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.591245 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.591194    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcvk\" (UniqueName: \"kubernetes.io/projected/57f24d2a-f57a-4a1a-93a5-180bafac42fb-kube-api-access-spcvk\") pod \"downloads-76f99584f4-zfs8p\" (UID: \"57f24d2a-f57a-4a1a-93a5-180bafac42fb\") " pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:35.676296 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.676276    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:35.680372 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-bound-sa-token\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680428 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-trusted-ca\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680428 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndgl\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-kube-api-access-bndgl\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-installation-pull-secrets\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6c46-0298-4d37-bab0-56acfc42d99a-ca-trust-extracted\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-tls\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680680 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-image-registry-private-configuration\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.680680 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.680627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-certificates\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.689995 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.689971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6c46-0298-4d37-bab0-56acfc42d99a-ca-trust-extracted\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.690586 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.690564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-certificates\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.690741 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.690717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f6c46-0298-4d37-bab0-56acfc42d99a-trusted-ca\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.692209 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.692156    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-bound-sa-token\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.692318 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.692215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-image-registry-private-configuration\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.692527 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.692505    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-registry-tls\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.692586 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.692511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee3f6c46-0298-4d37-bab0-56acfc42d99a-installation-pull-secrets\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.692625 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.692595    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndgl\" (UniqueName: \"kubernetes.io/projected/ee3f6c46-0298-4d37-bab0-56acfc42d99a-kube-api-access-bndgl\") pod \"image-registry-64cb7cc9d7-gdr5h\" (UID: \"ee3f6c46-0298-4d37-bab0-56acfc42d99a\") " pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.769153 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.769127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:35.774604 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.774581    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6hknm"
Sep 10 23:37:35.966611 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:35.966589    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zfs8p"]
Sep 10 23:37:35.968971 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:35.968943    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f24d2a_f57a_4a1a_93a5_180bafac42fb.slice/crio-734582d5f53603730ab56dbd4c54da81d4ebcb7569b4cc81f31247b98ad9d4ed WatchSource:0}: Error finding container 734582d5f53603730ab56dbd4c54da81d4ebcb7569b4cc81f31247b98ad9d4ed: Status 404 returned error can't find the container with id 734582d5f53603730ab56dbd4c54da81d4ebcb7569b4cc81f31247b98ad9d4ed
Sep 10 23:37:36.068915 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.068894    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"]
Sep 10 23:37:36.071155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:36.071127    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3f6c46_0298_4d37_bab0_56acfc42d99a.slice/crio-ad7c72d9856226a131dd860a6e8afbcd13a3124da1f2f2e13a2fb21a64cdcb98 WatchSource:0}: Error finding container ad7c72d9856226a131dd860a6e8afbcd13a3124da1f2f2e13a2fb21a64cdcb98: Status 404 returned error can't find the container with id ad7c72d9856226a131dd860a6e8afbcd13a3124da1f2f2e13a2fb21a64cdcb98
Sep 10 23:37:36.887885 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.887436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h" event={"ID":"ee3f6c46-0298-4d37-bab0-56acfc42d99a","Type":"ContainerStarted","Data":"472b2e0d3de5d62c41e39d9c9ba6ce3ebc55cda021b8ce455e9dd9e74d1c6213"}
Sep 10 23:37:36.887885 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.887492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h" event={"ID":"ee3f6c46-0298-4d37-bab0-56acfc42d99a","Type":"ContainerStarted","Data":"ad7c72d9856226a131dd860a6e8afbcd13a3124da1f2f2e13a2fb21a64cdcb98"}
Sep 10 23:37:36.887885 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.887552    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:36.888885 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.888853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zfs8p" event={"ID":"57f24d2a-f57a-4a1a-93a5-180bafac42fb","Type":"ContainerStarted","Data":"734582d5f53603730ab56dbd4c54da81d4ebcb7569b4cc81f31247b98ad9d4ed"}
Sep 10 23:37:36.919244 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:36.919191    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h" podStartSLOduration=1.919171783 podStartE2EDuration="1.919171783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:36.916998702 +0000 UTC m=+60.746825949" watchObservedRunningTime="2026-09-10 23:37:36.919171783 +0000 UTC m=+60.748999002"
Sep 10 23:37:39.866784 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:39.866736    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9jhd"
Sep 10 23:37:42.333438 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.333404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:42.333948 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.333470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:42.335741 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.335715    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:37:42.336269 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.336246    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:37:42.346451 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.346426    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53618be8-c85c-4f61-acb4-1e738be631f0-original-pull-secret\") pod \"global-pull-secret-syncer-528jm\" (UID: \"53618be8-c85c-4f61-acb4-1e738be631f0\") " pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:42.346580 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.346434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f16ef-b837-4aac-b339-f93860a0053a-metrics-certs\") pod \"network-metrics-daemon-msf67\" (UID: \"906f16ef-b837-4aac-b339-f93860a0053a\") " pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:42.433950 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.433914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:42.436401 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.436371    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:37:42.446935 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.446916    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:37:42.458116 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.458084    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68dz\" (UniqueName: \"kubernetes.io/projected/6bc07400-5b6d-4751-8c45-2b5db79f91dc-kube-api-access-w68dz\") pod \"network-check-target-h28ql\" (UID: \"6bc07400-5b6d-4751-8c45-2b5db79f91dc\") " pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:42.519089 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.519058    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\""
Sep 10 23:37:42.527271 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.527252    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-msf67"
Sep 10 23:37:42.621477 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.621278    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\""
Sep 10 23:37:42.629919 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.629896    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:42.631222 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.631176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-528jm"
Sep 10 23:37:42.831685 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.831655    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-msf67"]
Sep 10 23:37:42.835047 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:42.835022    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906f16ef_b837_4aac_b339_f93860a0053a.slice/crio-9c45dbb7ddf5c3d756a7e129a5657ae431e37d040c3a01958b7ca41e8249a07d WatchSource:0}: Error finding container 9c45dbb7ddf5c3d756a7e129a5657ae431e37d040c3a01958b7ca41e8249a07d: Status 404 returned error can't find the container with id 9c45dbb7ddf5c3d756a7e129a5657ae431e37d040c3a01958b7ca41e8249a07d
Sep 10 23:37:42.920309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.920232    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msf67" event={"ID":"906f16ef-b837-4aac-b339-f93860a0053a","Type":"ContainerStarted","Data":"9c45dbb7ddf5c3d756a7e129a5657ae431e37d040c3a01958b7ca41e8249a07d"}
Sep 10 23:37:42.934287 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.934260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h28ql"]
Sep 10 23:37:42.937283 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:42.937254    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc07400_5b6d_4751_8c45_2b5db79f91dc.slice/crio-e78b124d83bf5f8ba1e31d6b81760ddaf4186425b24465793c751dbd0cac8b4f WatchSource:0}: Error finding container e78b124d83bf5f8ba1e31d6b81760ddaf4186425b24465793c751dbd0cac8b4f: Status 404 returned error can't find the container with id e78b124d83bf5f8ba1e31d6b81760ddaf4186425b24465793c751dbd0cac8b4f
Sep 10 23:37:42.958432 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:42.958408    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-528jm"]
Sep 10 23:37:42.961779 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:42.961737    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53618be8_c85c_4f61_acb4_1e738be631f0.slice/crio-6083721e698cd4354832fa324a4ba3b6a566d097f4ad530c6f3c3c5009578b59 WatchSource:0}: Error finding container 6083721e698cd4354832fa324a4ba3b6a566d097f4ad530c6f3c3c5009578b59: Status 404 returned error can't find the container with id 6083721e698cd4354832fa324a4ba3b6a566d097f4ad530c6f3c3c5009578b59
Sep 10 23:37:43.925830 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:43.925780    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-528jm" event={"ID":"53618be8-c85c-4f61-acb4-1e738be631f0","Type":"ContainerStarted","Data":"6083721e698cd4354832fa324a4ba3b6a566d097f4ad530c6f3c3c5009578b59"}
Sep 10 23:37:43.927928 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:43.927903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h28ql" event={"ID":"6bc07400-5b6d-4751-8c45-2b5db79f91dc","Type":"ContainerStarted","Data":"e78b124d83bf5f8ba1e31d6b81760ddaf4186425b24465793c751dbd0cac8b4f"}
Sep 10 23:37:44.935790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:44.935419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msf67" event={"ID":"906f16ef-b837-4aac-b339-f93860a0053a","Type":"ContainerStarted","Data":"c1c5b36b9df94a9546a7a54747410e9a28f6230937cef93679c7fe288419c393"}
Sep 10 23:37:44.935790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:44.935492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-msf67" event={"ID":"906f16ef-b837-4aac-b339-f93860a0053a","Type":"ContainerStarted","Data":"1e5218eee660c01027b55632fee00fc4d76dd6903dbed61466c5508d9a3dcd8f"}
Sep 10 23:37:44.953199 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:44.953141    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-msf67" podStartSLOduration=67.945597499 podStartE2EDuration="1m8.953119658s" totalImagesPullingTime="1.007522159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:44.951498715 +0000 UTC m=+68.781325938" watchObservedRunningTime="2026-09-10 23:37:44.953119658 +0000 UTC m=+68.782946881"
Sep 10 23:37:45.376001 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:45.375824    2574 patch_prober.go:28] interesting pod/image-registry-f5f446cc6-f9gb9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 23:37:45.376001 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:45.375905    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:37:51.659112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.658987    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ls96s"]
Sep 10 23:37:51.662554 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.662526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.665025    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.665322    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.665511    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.665645    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.665971    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k96nt\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.666175    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:37:51.666456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.666327    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cj8p\" (UniqueName: \"kubernetes.io/projected/b1147a04-fea8-4a81-b680-82edc1af9d05-kube-api-access-6cj8p\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-textfile\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815605    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-sys\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.815677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-metrics-client-ca\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.816112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-root\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.816112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-wtmp\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.816112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.815855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-tls\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-sys\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-metrics-client-ca\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-root\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-wtmp\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-tls\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cj8p\" (UniqueName: \"kubernetes.io/projected/b1147a04-fea8-4a81-b680-82edc1af9d05-kube-api-access-6cj8p\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.917399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-textfile\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918067 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-root\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918067 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-sys\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918067 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-textfile\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918067 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.917836    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-wtmp\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918371 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.918228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-metrics-client-ca\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.918371 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.918275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.919915 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.919883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.920415 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.920393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1147a04-fea8-4a81-b680-82edc1af9d05-node-exporter-tls\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.927855 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.927827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cj8p\" (UniqueName: \"kubernetes.io/projected/b1147a04-fea8-4a81-b680-82edc1af9d05-kube-api-access-6cj8p\") pod \"node-exporter-ls96s\" (UID: \"b1147a04-fea8-4a81-b680-82edc1af9d05\") " pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:51.975402 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:51.975378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ls96s"
Sep 10 23:37:55.000227 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:54.998260    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1147a04_fea8_4a81_b680_82edc1af9d05.slice/crio-f400bd7765bcf557d407bfdcae7e8683a2ea2baed6b88695b4271177c38c8288 WatchSource:0}: Error finding container f400bd7765bcf557d407bfdcae7e8683a2ea2baed6b88695b4271177c38c8288: Status 404 returned error can't find the container with id f400bd7765bcf557d407bfdcae7e8683a2ea2baed6b88695b4271177c38c8288
Sep 10 23:37:55.373800 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.373503    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:37:55.981911 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.981869    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zfs8p" event={"ID":"57f24d2a-f57a-4a1a-93a5-180bafac42fb","Type":"ContainerStarted","Data":"b0d675af0efa115acbdab1c9741a0778b8f1718d7be343019b779b4d22ed4e69"}
Sep 10 23:37:55.982166 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.982104    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:55.983644 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.983601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-528jm" event={"ID":"53618be8-c85c-4f61-acb4-1e738be631f0","Type":"ContainerStarted","Data":"04dc6266989a8be335935c0d07a333fa28899e2e498bba60a4ec3367abb91edf"}
Sep 10 23:37:55.985540 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.985450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h28ql" event={"ID":"6bc07400-5b6d-4751-8c45-2b5db79f91dc","Type":"ContainerStarted","Data":"82bcbec4c553033a43022281e99f70ab1c021ad1492621f76e3be86ee344f173"}
Sep 10 23:37:55.985711 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.985682    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:37:55.987661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.987439    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls96s" event={"ID":"b1147a04-fea8-4a81-b680-82edc1af9d05","Type":"ContainerStarted","Data":"d4755c3df50be5f75464bf9a1b59917ffae21786e822d8f82442ae46323d77c0"}
Sep 10 23:37:55.987661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.987486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls96s" event={"ID":"b1147a04-fea8-4a81-b680-82edc1af9d05","Type":"ContainerStarted","Data":"f400bd7765bcf557d407bfdcae7e8683a2ea2baed6b88695b4271177c38c8288"}
Sep 10 23:37:55.991669 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:55.991648    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-zfs8p"
Sep 10 23:37:56.002754 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:56.002699    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-zfs8p" podStartSLOduration=1.9150678349999999 podStartE2EDuration="21.002681046s" totalImagesPullingTime="19.087613211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:56.002207145 +0000 UTC m=+79.832034366" watchObservedRunningTime="2026-09-10 23:37:56.002681046 +0000 UTC m=+79.832508265"
Sep 10 23:37:56.019187 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:56.019130    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h28ql" podStartSLOduration=67.963742968 podStartE2EDuration="1m20.019109437s" totalImagesPullingTime="12.055366469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:56.01897723 +0000 UTC m=+79.848804449" watchObservedRunningTime="2026-09-10 23:37:56.019109437 +0000 UTC m=+79.848936655"
Sep 10 23:37:56.078758 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:56.078703    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-528jm" podStartSLOduration=68.034537381 podStartE2EDuration="1m20.078680859s" totalImagesPullingTime="12.044143478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:56.07767435 +0000 UTC m=+79.907501570" watchObservedRunningTime="2026-09-10 23:37:56.078680859 +0000 UTC m=+79.908508081"
Sep 10 23:37:56.994146 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:56.994107    2574 generic.go:356] "Generic (PLEG): container finished" podID="b1147a04-fea8-4a81-b680-82edc1af9d05" containerID="d4755c3df50be5f75464bf9a1b59917ffae21786e822d8f82442ae46323d77c0" exitCode=0
Sep 10 23:37:56.994337 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:56.994198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls96s" event={"ID":"b1147a04-fea8-4a81-b680-82edc1af9d05","Type":"ContainerDied","Data":"d4755c3df50be5f75464bf9a1b59917ffae21786e822d8f82442ae46323d77c0"}
Sep 10 23:37:57.898427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:57.898393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64cb7cc9d7-gdr5h"
Sep 10 23:37:57.983735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:57.983703    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:57.999595 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:57.999566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.000894 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.000857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls96s" event={"ID":"b1147a04-fea8-4a81-b680-82edc1af9d05","Type":"ContainerStarted","Data":"b3961607e10592afe51a0a6660edd96d492dae6340399c5d20f2993c217a40a0"}
Sep 10 23:37:58.001030 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.000920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls96s" event={"ID":"b1147a04-fea8-4a81-b680-82edc1af9d05","Type":"ContainerStarted","Data":"102174a60290864185e6b45e939e3c7e7015678f957ac1114fbe2f5da2d8f85c"}
Sep 10 23:37:58.001920 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.001901    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 23:37:58.003225 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1rqji69a4cp22\""
Sep 10 23:37:58.003328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003254    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 23:37:58.003328 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003269    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 23:37:58.003441 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003258    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 23:37:58.003441 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003407    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 23:37:58.003737 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003665    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 23:37:58.003737 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003689    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 23:37:58.003888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003744    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 23:37:58.003888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.003866    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 23:37:58.004749 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.004729    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 23:37:58.005227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.005205    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r772v\""
Sep 10 23:37:58.005386 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.005371    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 23:37:58.005732 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.005716    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 23:37:58.006734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.006612    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 23:37:58.011293 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.011272    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 23:37:58.012847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.012828    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 23:37:58.016036 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.016014    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:58.067685 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.067630    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ls96s" podStartSLOduration=6.113945185 podStartE2EDuration="7.067610442s" totalImagesPullingTime="875.04717ms" totalInitContainerRuntime="78.618087ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:58.066850156 +0000 UTC m=+81.896677377" watchObservedRunningTime="2026-09-10 23:37:58.067610442 +0000 UTC m=+81.897437717"
Sep 10 23:37:58.073234 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073387 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073387 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073511 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfttv\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-kube-api-access-nfttv\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.073829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074015 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074015 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074015 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074015 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.073996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074210 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.074021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.074210 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.074166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.174790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.174790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.174790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174861    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.174972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.175003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175702 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.175497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.175855 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.175831    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfttv\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-kube-api-access-nfttv\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.176963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.176490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.181598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.178170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.181598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.178629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.181598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.178664    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.181598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.178745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.181598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.180927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.182463 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.182440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.183363 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.183335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.186493 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.186108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.188824 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.187171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.189422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.189324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.190886 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.189850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.190886 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.190078    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfttv\" (UniqueName: \"kubernetes.io/projected/e2c38600-cd01-4db0-ad4d-51afa0c2304a-kube-api-access-nfttv\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.191247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.191223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c38600-cd01-4db0-ad4d-51afa0c2304a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.191681 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.191662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c38600-cd01-4db0-ad4d-51afa0c2304a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.193454 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.193405    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.194078 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.194045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.194584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.194563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c38600-cd01-4db0-ad4d-51afa0c2304a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2c38600-cd01-4db0-ad4d-51afa0c2304a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.312132 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.312092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:37:58.625685 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:58.625660    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:37:58.627552 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:37:58.627525    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c38600_cd01_4db0_ad4d_51afa0c2304a.slice/crio-686f32c07d6f89f210aee5ceee541d7c7288feac6fe712100db49731bb1a7ed0 WatchSource:0}: Error finding container 686f32c07d6f89f210aee5ceee541d7c7288feac6fe712100db49731bb1a7ed0: Status 404 returned error can't find the container with id 686f32c07d6f89f210aee5ceee541d7c7288feac6fe712100db49731bb1a7ed0
Sep 10 23:37:59.008935 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:37:59.008849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"686f32c07d6f89f210aee5ceee541d7c7288feac6fe712100db49731bb1a7ed0"}
Sep 10 23:38:00.386046 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.385994    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerName="registry" containerID="cri-o://a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c" gracePeriod=30
Sep 10 23:38:00.681442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.681415    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:38:00.802501 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802463    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m44qs\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.802501 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.802755 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.802755 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802568    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.802755 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802618    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.802755 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.802672    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.803047 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803015    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.803126 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803039    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:00.803126 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") pod \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\" (UID: \"cc2a8555-25a2-491d-bcd1-8a8794de15c3\") "
Sep 10 23:38:00.803126 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803093    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:38:00.803622 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803359    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-trusted-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.803622 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.803384    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-certificates\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.805610 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.805562    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:00.805610 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.805575    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs" (OuterVolumeSpecName: "kube-api-access-m44qs") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "kube-api-access-m44qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:00.805785 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.805719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:00.805851 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.805825    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:38:00.805910 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.805872    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:38:00.816828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.816794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cc2a8555-25a2-491d-bcd1-8a8794de15c3" (UID: "cc2a8555-25a2-491d-bcd1-8a8794de15c3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:38:00.904637 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904563    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-installation-pull-secrets\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.904637 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904601    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-registry-tls\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.904637 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904618    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44qs\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-kube-api-access-m44qs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.904637 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904632    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc2a8555-25a2-491d-bcd1-8a8794de15c3-bound-sa-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.904959 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904647    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc2a8555-25a2-491d-bcd1-8a8794de15c3-ca-trust-extracted\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:00.904959 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:00.904661    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc2a8555-25a2-491d-bcd1-8a8794de15c3-image-registry-private-configuration\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:38:01.019094 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.019053    2574 generic.go:356] "Generic (PLEG): container finished" podID="e2c38600-cd01-4db0-ad4d-51afa0c2304a" containerID="ef6f943b148a40deee764de1b800456f516557fc67d3de80c9504ab7eeaba436" exitCode=0
Sep 10 23:38:01.019276 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.019161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerDied","Data":"ef6f943b148a40deee764de1b800456f516557fc67d3de80c9504ab7eeaba436"}
Sep 10 23:38:01.020756 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.020727    2574 generic.go:356] "Generic (PLEG): container finished" podID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerID="a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c" exitCode=0
Sep 10 23:38:01.020893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.020787    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9"
Sep 10 23:38:01.020893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.020807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" event={"ID":"cc2a8555-25a2-491d-bcd1-8a8794de15c3","Type":"ContainerDied","Data":"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"}
Sep 10 23:38:01.020893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.020852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5f446cc6-f9gb9" event={"ID":"cc2a8555-25a2-491d-bcd1-8a8794de15c3","Type":"ContainerDied","Data":"ea2b06d30fcbd937c2ed7fd27f5602b1a084d374a99df6d8d245f606ddfbf055"}
Sep 10 23:38:01.020893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.020882    2574 scope.go:117] "RemoveContainer" containerID="a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"
Sep 10 23:38:01.032408 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.032377    2574 scope.go:117] "RemoveContainer" containerID="a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"
Sep 10 23:38:01.032793 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:38:01.032744    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c\": container with ID starting with a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c not found: ID does not exist" containerID="a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"
Sep 10 23:38:01.032893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.032800    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c"} err="failed to get container status \"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c\": rpc error: code = NotFound desc = could not find container \"a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c\": container with ID starting with a60f4142e426014f357ed23992439475a013310bc1947334b7c2d5092678b98c not found: ID does not exist"
Sep 10 23:38:01.062491 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.062456    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:38:01.066373 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:01.066346    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f5f446cc6-f9gb9"]
Sep 10 23:38:02.615714 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:02.615667    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" path="/var/lib/kubelet/pods/cc2a8555-25a2-491d-bcd1-8a8794de15c3/volumes"
Sep 10 23:38:05.040571 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:05.040539    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"14fd55f81f37b875691937fd6c027753f212fcca937a0c97887c5b105b79c845"}
Sep 10 23:38:05.040995 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:05.040604    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"c452578abd3244e850ba5aca96cef1eecd06262c4668d31f11f4007884a156bc"}
Sep 10 23:38:07.050280 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:07.050251    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"71e92b514de064043fec0aa9b9aff6e1feda3798d9bd05efa8b85dffb44ea3d0"}
Sep 10 23:38:08.057349 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:08.057314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"bfbe703c278389b7f4582f48f9c4ad186a9850d63dcd8db32b2513181ef32786"}
Sep 10 23:38:08.057844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:08.057378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"c195024d33698bdfa346e8b5b511061040d002504626d33c4bdeffe3b35c4da0"}
Sep 10 23:38:08.057844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:08.057408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2c38600-cd01-4db0-ad4d-51afa0c2304a","Type":"ContainerStarted","Data":"38c9af832a56d1a03f0ec2bc4a0a9de872aa131ee9bada408e02f5c21400a7d1"}
Sep 10 23:38:08.092594 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:08.092524    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.11682668 podStartE2EDuration="11.092505759s" totalImagesPullingTime="6.946651302s" totalInitContainerRuntime="29.027777ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:08.091456378 +0000 UTC m=+91.921283620" watchObservedRunningTime="2026-09-10 23:38:08.092505759 +0000 UTC m=+91.922332979"
Sep 10 23:38:08.312600 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:08.312509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:26.997407 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:26.997376    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h28ql"
Sep 10 23:38:58.312546 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:58.312503    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:58.332701 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:58.332672    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:38:59.250426 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:38:59.250398    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:41:09.182290 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.182201    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"]
Sep 10 23:41:09.182890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.182568    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerName="registry"
Sep 10 23:41:09.182890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.182584    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerName="registry"
Sep 10 23:41:09.182890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.182656    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2a8555-25a2-491d-bcd1-8a8794de15c3" containerName="registry"
Sep 10 23:41:09.184603 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.184582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.186895 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.186874    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 23:41:09.187052 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.187037    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xxcdp\""
Sep 10 23:41:09.187114 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.187060    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:41:09.194974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.194953    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"]
Sep 10 23:41:09.252438 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.252408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.252578 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.252475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncnr\" (UniqueName: \"kubernetes.io/projected/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-kube-api-access-4ncnr\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.353862 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.353828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.354017 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.353912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncnr\" (UniqueName: \"kubernetes.io/projected/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-kube-api-access-4ncnr\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.354295 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.354277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.362872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.362847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncnr\" (UniqueName: \"kubernetes.io/projected/d0131d64-a7e2-4ef5-b7f4-fec464243cfa-kube-api-access-4ncnr\") pod \"openshift-lws-operator-fd8ccff4c-ssf8q\" (UID: \"d0131d64-a7e2-4ef5-b7f4-fec464243cfa\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.503410 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.503327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"
Sep 10 23:41:09.794662 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:09.794593    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q"]
Sep 10 23:41:09.797728 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:41:09.797701    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0131d64_a7e2_4ef5_b7f4_fec464243cfa.slice/crio-d420d845cf98890f4861a61c2036459be889285a6df740993ce6fbf699103fa1 WatchSource:0}: Error finding container d420d845cf98890f4861a61c2036459be889285a6df740993ce6fbf699103fa1: Status 404 returned error can't find the container with id d420d845cf98890f4861a61c2036459be889285a6df740993ce6fbf699103fa1
Sep 10 23:41:10.669531 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:10.669472    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q" event={"ID":"d0131d64-a7e2-4ef5-b7f4-fec464243cfa","Type":"ContainerStarted","Data":"d420d845cf98890f4861a61c2036459be889285a6df740993ce6fbf699103fa1"}
Sep 10 23:41:12.679456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:12.679424    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q" event={"ID":"d0131d64-a7e2-4ef5-b7f4-fec464243cfa","Type":"ContainerStarted","Data":"09bc0b37fc8f42f1aaf5d5860300535f3a6cd1743c315f092cfaad3030982f88"}
Sep 10 23:41:12.694320 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:12.694275    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssf8q" podStartSLOduration=1.400708501 podStartE2EDuration="3.694255258s" totalImagesPullingTime="2.293546757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:12.694004678 +0000 UTC m=+276.523831898" watchObservedRunningTime="2026-09-10 23:41:12.694255258 +0000 UTC m=+276.524082477"
Sep 10 23:41:32.101078 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.101042    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"]
Sep 10 23:41:32.103222 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.103206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.106530 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.106504    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:41:32.106713 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.106700    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:41:32.106784 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.106754    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:41:32.107071 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.107047    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:41:32.107207 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.107113    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b879w\""
Sep 10 23:41:32.120542 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.120516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"]
Sep 10 23:41:32.223561 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.223528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.223726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.223570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q452n\" (UniqueName: \"kubernetes.io/projected/23897cad-399e-4809-be7e-2d370ef1e897-kube-api-access-q452n\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.223726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.223601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.223726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.223640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.223726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.223698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324242 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q452n\" (UniqueName: \"kubernetes.io/projected/23897cad-399e-4809-be7e-2d370ef1e897-kube-api-access-q452n\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324242 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324242 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.324522 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.324499    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.326284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.326253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23897cad-399e-4809-be7e-2d370ef1e897-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.326449 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.326428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.326599 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.326579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/23897cad-399e-4809-be7e-2d370ef1e897-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.335938 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.335917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q452n\" (UniqueName: \"kubernetes.io/projected/23897cad-399e-4809-be7e-2d370ef1e897-kube-api-access-q452n\") pod \"opendatahub-operator-controller-manager-55d55f8464-wpbs2\" (UID: \"23897cad-399e-4809-be7e-2d370ef1e897\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.413670 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.413609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:32.693172 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.692940    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"]
Sep 10 23:41:32.695935 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:41:32.695909    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23897cad_399e_4809_be7e_2d370ef1e897.slice/crio-275761eddaf68f79f56bfe890807958e9ed6e34c331fde7c781232e545136cf8 WatchSource:0}: Error finding container 275761eddaf68f79f56bfe890807958e9ed6e34c331fde7c781232e545136cf8: Status 404 returned error can't find the container with id 275761eddaf68f79f56bfe890807958e9ed6e34c331fde7c781232e545136cf8
Sep 10 23:41:32.745864 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:32.745835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2" event={"ID":"23897cad-399e-4809-be7e-2d370ef1e897","Type":"ContainerStarted","Data":"275761eddaf68f79f56bfe890807958e9ed6e34c331fde7c781232e545136cf8"}
Sep 10 23:41:35.760222 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:35.760187    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2" event={"ID":"23897cad-399e-4809-be7e-2d370ef1e897","Type":"ContainerStarted","Data":"1431f4c1b080593af646ca0f74dec9119dc8797f8b750bf8cdd0295be17321b9"}
Sep 10 23:41:36.532793 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:36.532750    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:41:36.765265 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:36.765230    2574 generic.go:356] "Generic (PLEG): container finished" podID="23897cad-399e-4809-be7e-2d370ef1e897" containerID="1431f4c1b080593af646ca0f74dec9119dc8797f8b750bf8cdd0295be17321b9" exitCode=0
Sep 10 23:41:36.765756 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:36.765297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2" event={"ID":"23897cad-399e-4809-be7e-2d370ef1e897","Type":"ContainerDied","Data":"1431f4c1b080593af646ca0f74dec9119dc8797f8b750bf8cdd0295be17321b9"}
Sep 10 23:41:37.771903 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:37.771872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2" event={"ID":"23897cad-399e-4809-be7e-2d370ef1e897","Type":"ContainerStarted","Data":"be3ce9c3ba667e0c630d4605a786917cda3bf59e39ddceec25f9c0c89b7cae83"}
Sep 10 23:41:37.772279 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:37.771997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:37.794866 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:37.794817    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2" podStartSLOduration=1.634411638 podStartE2EDuration="5.7948007s" totalImagesPullingTime="3.123816492s" totalInitContainerRuntime="1.03657257s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:37.792800792 +0000 UTC m=+301.622628008" watchObservedRunningTime="2026-09-10 23:41:37.7948007 +0000 UTC m=+301.624627981"
Sep 10 23:41:48.777406 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:48.777376    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-wpbs2"
Sep 10 23:41:52.133491 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.133457    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"]
Sep 10 23:41:52.140316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.140295    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.143814 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.143794    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 23:41:52.143943 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.143795    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 23:41:52.143943 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.143815    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:41:52.144166 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.144151    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:41:52.144310 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.144182    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-npjkn\""
Sep 10 23:41:52.162459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.162441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b099978-183b-4d8c-9e00-4b004e5cf76c-tls-certs\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.162541 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.162472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b099978-183b-4d8c-9e00-4b004e5cf76c-tmp\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.162541 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.162494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqb28\" (UniqueName: \"kubernetes.io/projected/1b099978-183b-4d8c-9e00-4b004e5cf76c-kube-api-access-tqb28\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.176087 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.176066    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"]
Sep 10 23:41:52.263276 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.263252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b099978-183b-4d8c-9e00-4b004e5cf76c-tls-certs\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.263402 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.263289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b099978-183b-4d8c-9e00-4b004e5cf76c-tmp\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.263402 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.263313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqb28\" (UniqueName: \"kubernetes.io/projected/1b099978-183b-4d8c-9e00-4b004e5cf76c-kube-api-access-tqb28\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.265696 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.265671    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b099978-183b-4d8c-9e00-4b004e5cf76c-tmp\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.265893 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.265873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b099978-183b-4d8c-9e00-4b004e5cf76c-tls-certs\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.311207 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.311176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqb28\" (UniqueName: \"kubernetes.io/projected/1b099978-183b-4d8c-9e00-4b004e5cf76c-kube-api-access-tqb28\") pod \"kube-auth-proxy-55f89b848f-f2qxh\" (UID: \"1b099978-183b-4d8c-9e00-4b004e5cf76c\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.449539 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.449469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"
Sep 10 23:41:52.742906 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.742821    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh"]
Sep 10 23:41:52.746908 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:41:52.746886    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b099978_183b_4d8c_9e00_4b004e5cf76c.slice/crio-19f10c029a26119d885c797684a533f5d37afc74d3360746c35739a408beab08 WatchSource:0}: Error finding container 19f10c029a26119d885c797684a533f5d37afc74d3360746c35739a408beab08: Status 404 returned error can't find the container with id 19f10c029a26119d885c797684a533f5d37afc74d3360746c35739a408beab08
Sep 10 23:41:52.748871 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.748842    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:41:52.824430 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:52.824399    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh" event={"ID":"1b099978-183b-4d8c-9e00-4b004e5cf76c","Type":"ContainerStarted","Data":"19f10c029a26119d885c797684a533f5d37afc74d3360746c35739a408beab08"}
Sep 10 23:41:55.840349 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:55.840315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh" event={"ID":"1b099978-183b-4d8c-9e00-4b004e5cf76c","Type":"ContainerStarted","Data":"e29c5f957c9d1475d250cbd3d817a0c3c1a8c6021f96a20d22cc157083e34390"}
Sep 10 23:41:55.855945 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:41:55.855900    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55f89b848f-f2qxh" podStartSLOduration=1.3314455760000001 podStartE2EDuration="3.855884953s" totalImagesPullingTime="2.524439377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:55.854681222 +0000 UTC m=+319.684508442" watchObservedRunningTime="2026-09-10 23:41:55.855884953 +0000 UTC m=+319.685712173"
Sep 10 23:42:00.151886 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.151845    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:00.154124 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.154102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.156549 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.156529    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 23:42:00.156663 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.156636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rtqfv\""
Sep 10 23:42:00.167535 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.167514    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:00.216665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.216631    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"]
Sep 10 23:42:00.219044 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.219025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.220729 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220849 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmnb\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.220963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.221054 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.220990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.221054 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.221037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.231984 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.231961    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"]
Sep 10 23:42:00.321896 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.321896 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.321998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hr9b\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-kube-api-access-9hr9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmnb\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322142 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322512    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.322671 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322531    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.323033 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322714    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.323033 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.323033 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.322911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.324619 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.324596    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.324891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.324873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.331810 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.331789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmnb\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.332279 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.332259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctpqqp\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.423361 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423361 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hr9b\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-kube-api-access-9hr9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423361 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423636 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423937 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.423937 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.423922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.424230 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.424188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.424230 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.424196    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.424368 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.424230    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.425741 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.425720    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.426086 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.426065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.432034 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.432011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hr9b\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-kube-api-access-9hr9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.432338 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.432320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1eccc1-a13d-4628-9d57-85eebaffe30f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p\" (UID: \"2d1eccc1-a13d-4628-9d57-85eebaffe30f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.467423 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.467391    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:00.529300 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.529265    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:00.741724 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.741698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:00.743749 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:42:00.743722    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6ac678_6217_4eb3_bb5c_fd86d14d89e3.slice/crio-7ae2c8ccab9149747f08a85a45ecd114da1b1bd2fa4749179a305b73732b4d95 WatchSource:0}: Error finding container 7ae2c8ccab9149747f08a85a45ecd114da1b1bd2fa4749179a305b73732b4d95: Status 404 returned error can't find the container with id 7ae2c8ccab9149747f08a85a45ecd114da1b1bd2fa4749179a305b73732b4d95
Sep 10 23:42:00.801622 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.801589    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"]
Sep 10 23:42:00.804296 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:42:00.804267    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1eccc1_a13d_4628_9d57_85eebaffe30f.slice/crio-b042801ba58beaa5a5cc17198073cc7ed58f0b5e89b74fabe2967999faf1bf8b WatchSource:0}: Error finding container b042801ba58beaa5a5cc17198073cc7ed58f0b5e89b74fabe2967999faf1bf8b: Status 404 returned error can't find the container with id b042801ba58beaa5a5cc17198073cc7ed58f0b5e89b74fabe2967999faf1bf8b
Sep 10 23:42:00.864884 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.864836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p" event={"ID":"2d1eccc1-a13d-4628-9d57-85eebaffe30f","Type":"ContainerStarted","Data":"b042801ba58beaa5a5cc17198073cc7ed58f0b5e89b74fabe2967999faf1bf8b"}
Sep 10 23:42:00.866117 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:00.866091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" event={"ID":"df6ac678-6217-4eb3-bb5c-fd86d14d89e3","Type":"ContainerStarted","Data":"7ae2c8ccab9149747f08a85a45ecd114da1b1bd2fa4749179a305b73732b4d95"}
Sep 10 23:42:03.772654 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.772376    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.772654 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.772502    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.772654 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.772546    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.784325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.784296    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.784414 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.784358    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.784414 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.784391    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:42:03.890629 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.890521    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p" event={"ID":"2d1eccc1-a13d-4628-9d57-85eebaffe30f","Type":"ContainerStarted","Data":"dc1509d52ad8ad739ecde43842ee70f2406d49d48863561cf6c52a13b442b9f7"}
Sep 10 23:42:03.892645 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.892592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" event={"ID":"df6ac678-6217-4eb3-bb5c-fd86d14d89e3","Type":"ContainerStarted","Data":"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"}
Sep 10 23:42:03.911144 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.911083    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p" podStartSLOduration=0.945312109 podStartE2EDuration="3.911062943s" totalImagesPullingTime="2.965750834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:03.908882092 +0000 UTC m=+327.738709312" watchObservedRunningTime="2026-09-10 23:42:03.911062943 +0000 UTC m=+327.740890162"
Sep 10 23:42:03.928899 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:03.928696    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" podStartSLOduration=0.890361715 podStartE2EDuration="3.928677593s" totalImagesPullingTime="3.038315878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:03.927302077 +0000 UTC m=+327.757129301" watchObservedRunningTime="2026-09-10 23:42:03.928677593 +0000 UTC m=+327.758504813"
Sep 10 23:42:04.467846 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.467811    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:04.469479 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.469445    2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Sep 10 23:42:04.469632 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.469511    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Sep 10 23:42:04.530527 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.530491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:04.535994 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.535968    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:04.896981 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.896949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:04.897955 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.897939    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p"
Sep 10 23:42:04.968326 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:04.968292    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:05.468246 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:05.468214    2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Sep 10 23:42:05.468426 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:05.468266    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Sep 10 23:42:06.468760 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:06.468726    2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Sep 10 23:42:06.469151 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:06.468818    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Sep 10 23:42:06.904452 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:06.904408    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy" containerID="cri-o://827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61" gracePeriod=30
Sep 10 23:42:09.730832 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.730798    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"]
Sep 10 23:42:09.738001 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.737977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.741148 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.741124    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m2crj\""
Sep 10 23:42:09.741292 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.741194    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 23:42:09.741292 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.741197    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 23:42:09.741413 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.741301    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 23:42:09.744115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.744095    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"]
Sep 10 23:42:09.808162 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.808127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7518d434-5693-4795-9901-3c6691ebab73-manager-config\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.808162 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.808170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgtm5\" (UniqueName: \"kubernetes.io/projected/7518d434-5693-4795-9901-3c6691ebab73-kube-api-access-tgtm5\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.808379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.808213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.808379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.808235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.908854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.908819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7518d434-5693-4795-9901-3c6691ebab73-manager-config\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.908854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.908855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgtm5\" (UniqueName: \"kubernetes.io/projected/7518d434-5693-4795-9901-3c6691ebab73-kube-api-access-tgtm5\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.909103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.908886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.909103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.909004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.909542 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.909507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7518d434-5693-4795-9901-3c6691ebab73-manager-config\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.911472 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.911447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-metrics-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.911571 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.911447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7518d434-5693-4795-9901-3c6691ebab73-cert\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:09.916907 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:09.916883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgtm5\" (UniqueName: \"kubernetes.io/projected/7518d434-5693-4795-9901-3c6691ebab73-kube-api-access-tgtm5\") pod \"lws-controller-manager-66bf4bb994-4zhhq\" (UID: \"7518d434-5693-4795-9901-3c6691ebab73\") " pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:10.048131 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:10.048039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:10.319419 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:10.319396    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"]
Sep 10 23:42:10.321694 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:42:10.321667    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7518d434_5693_4795_9901_3c6691ebab73.slice/crio-22a413648c7cd7a9343fb65a9d9a7a7c1b6a1676c7aba6364a09049e3396f30c WatchSource:0}: Error finding container 22a413648c7cd7a9343fb65a9d9a7a7c1b6a1676c7aba6364a09049e3396f30c: Status 404 returned error can't find the container with id 22a413648c7cd7a9343fb65a9d9a7a7c1b6a1676c7aba6364a09049e3396f30c
Sep 10 23:42:10.923718 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:10.923684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq" event={"ID":"7518d434-5693-4795-9901-3c6691ebab73","Type":"ContainerStarted","Data":"22a413648c7cd7a9343fb65a9d9a7a7c1b6a1676c7aba6364a09049e3396f30c"}
Sep 10 23:42:11.929836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:11.929753    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq" event={"ID":"7518d434-5693-4795-9901-3c6691ebab73","Type":"ContainerStarted","Data":"1eee8b0345b61f31764675f2b195a2f4c23f14d258940b6a3c161e5666d87c75"}
Sep 10 23:42:11.930260 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:11.929878    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:42:11.949990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:11.949941    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq" podStartSLOduration=1.868787918 podStartE2EDuration="2.949925532s" totalImagesPullingTime="1.081137614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:11.948366013 +0000 UTC m=+335.778193235" watchObservedRunningTime="2026-09-10 23:42:11.949925532 +0000 UTC m=+335.779752796"
Sep 10 23:42:12.145063 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.145036    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:12.225890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.225818    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pmnb\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.225890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.225853    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226061 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.225899    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226061 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.225915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226061 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226012    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226061 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226041    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226089    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226247 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226176    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket\") pod \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\" (UID: \"df6ac678-6217-4eb3-bb5c-fd86d14d89e3\") "
Sep 10 23:42:12.226398 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226263    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:42:12.226452 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226395    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:42:12.226503 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226271    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:42:12.226503 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226484    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.226503 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226490    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:42:12.226653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226504    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.226653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.226627    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:42:12.228327 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.228297    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb" (OuterVolumeSpecName: "kube-api-access-7pmnb") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "kube-api-access-7pmnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:42:12.228451 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.228331    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:42:12.228451 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.228419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:42:12.228546 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.228468    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df6ac678-6217-4eb3-bb5c-fd86d14d89e3" (UID: "df6ac678-6217-4eb3-bb5c-fd86d14d89e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:42:12.327828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327800    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.327828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327824    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.327828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327833    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.328041 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327842    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmnb\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-kube-api-access-7pmnb\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.328041 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327851    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.328041 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327859    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.328041 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.327866    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6ac678-6217-4eb3-bb5c-fd86d14d89e3-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:42:12.937447 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.937346    2574 generic.go:356] "Generic (PLEG): container finished" podID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerID="827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61" exitCode=0
Sep 10 23:42:12.937447 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.937396    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"
Sep 10 23:42:12.937961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.937437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" event={"ID":"df6ac678-6217-4eb3-bb5c-fd86d14d89e3","Type":"ContainerDied","Data":"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"}
Sep 10 23:42:12.937961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.937503    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp" event={"ID":"df6ac678-6217-4eb3-bb5c-fd86d14d89e3","Type":"ContainerDied","Data":"7ae2c8ccab9149747f08a85a45ecd114da1b1bd2fa4749179a305b73732b4d95"}
Sep 10 23:42:12.937961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.937534    2574 scope.go:117] "RemoveContainer" containerID="827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"
Sep 10 23:42:12.946135 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.946115    2574 scope.go:117] "RemoveContainer" containerID="827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"
Sep 10 23:42:12.946450 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:42:12.946419    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61\": container with ID starting with 827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61 not found: ID does not exist" containerID="827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"
Sep 10 23:42:12.946522 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.946462    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61"} err="failed to get container status \"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61\": rpc error: code = NotFound desc = could not find container \"827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61\": container with ID starting with 827b61e82baa64e611dc5b21749b8d69d4b7313e4a1621976fd69ffd4db23a61 not found: ID does not exist"
Sep 10 23:42:12.990590 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:12.990566    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:13.003091 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:13.003068    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctpqqp"]
Sep 10 23:42:14.614717 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:14.614683    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" path="/var/lib/kubelet/pods/df6ac678-6217-4eb3-bb5c-fd86d14d89e3/volumes"
Sep 10 23:42:22.940325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:42:22.940295    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66bf4bb994-4zhhq"
Sep 10 23:43:33.333975 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.333939    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"]
Sep 10 23:43:33.334538 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.334310    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy"
Sep 10 23:43:33.334538 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.334329    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy"
Sep 10 23:43:33.334538 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.334381    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6ac678-6217-4eb3-bb5c-fd86d14d89e3" containerName="istio-proxy"
Sep 10 23:43:33.336128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.336112    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.338202 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.338182    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 23:43:33.338311 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.338294    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:43:33.338829 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.338812    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:43:33.339070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.339057    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fdzdl\""
Sep 10 23:43:33.343959 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.343939    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 23:43:33.349199 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.349178    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"]
Sep 10 23:43:33.464887 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.464854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.465026 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.464897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.465026 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.464924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc2w\" (UniqueName: \"kubernetes.io/projected/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-kube-api-access-2rc2w\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.565339 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.565296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.565482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.565352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.565482 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.565396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc2w\" (UniqueName: \"kubernetes.io/projected/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-kube-api-access-2rc2w\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.566070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.566045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.567712 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.567695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.572625 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.572607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc2w\" (UniqueName: \"kubernetes.io/projected/be189bf2-bdd6-47d2-ba14-c950dfbdf7c2-kube-api-access-2rc2w\") pod \"kuadrant-console-plugin-fbdd4f445-slrbk\" (UID: \"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.645691 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.645629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"
Sep 10 23:43:33.914626 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:33.914443    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk"]
Sep 10 23:43:33.917202 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:43:33.917164    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe189bf2_bdd6_47d2_ba14_c950dfbdf7c2.slice/crio-7485348fab56d6e5bd9ba985ac55a4f1643128c243246143fd001f1150376874 WatchSource:0}: Error finding container 7485348fab56d6e5bd9ba985ac55a4f1643128c243246143fd001f1150376874: Status 404 returned error can't find the container with id 7485348fab56d6e5bd9ba985ac55a4f1643128c243246143fd001f1150376874
Sep 10 23:43:34.266467 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:34.266388    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk" event={"ID":"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2","Type":"ContainerStarted","Data":"7485348fab56d6e5bd9ba985ac55a4f1643128c243246143fd001f1150376874"}
Sep 10 23:43:39.292574 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:39.292539    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk" event={"ID":"be189bf2-bdd6-47d2-ba14-c950dfbdf7c2","Type":"ContainerStarted","Data":"e9e842eb1a995876abc7f9c16dca5e349a776d05e731895608026c64f7474b06"}
Sep 10 23:43:39.312471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:39.312423    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-slrbk" podStartSLOduration=1.640127761 podStartE2EDuration="6.31240874s" totalImagesPullingTime="4.672280979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:39.310917066 +0000 UTC m=+423.140744285" watchObservedRunningTime="2026-09-10 23:43:39.31240874 +0000 UTC m=+423.142235960"
Sep 10 23:43:44.395990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.395957    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 10 23:43:44.399883 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.399864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.401989 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.401970    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5wwxc\""
Sep 10 23:43:44.427955 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.427925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 10 23:43:44.456897 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.456872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.457035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.456935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2622\" (UniqueName: \"kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.557653 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.557617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2622\" (UniqueName: \"kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.557797 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.557704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.558029 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.558011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.566074 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.566044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2622\" (UniqueName: \"kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cn8hx\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.709062 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.708995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:44.990018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:44.989947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 10 23:43:44.993398 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:43:44.993369    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7aef16_c34c_4a7f_8038_8912c23ad540.slice/crio-3b445e932ad82b80a93e21afd55e56b39e5d73870c34debdc797c2468909f0dc WatchSource:0}: Error finding container 3b445e932ad82b80a93e21afd55e56b39e5d73870c34debdc797c2468909f0dc: Status 404 returned error can't find the container with id 3b445e932ad82b80a93e21afd55e56b39e5d73870c34debdc797c2468909f0dc
Sep 10 23:43:45.317448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:45.317367    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" event={"ID":"fa7aef16-c34c-4a7f-8038-8912c23ad540","Type":"ContainerStarted","Data":"3b445e932ad82b80a93e21afd55e56b39e5d73870c34debdc797c2468909f0dc"}
Sep 10 23:43:49.338354 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:49.338314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" event={"ID":"fa7aef16-c34c-4a7f-8038-8912c23ad540","Type":"ContainerStarted","Data":"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"}
Sep 10 23:43:49.338815 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:49.338418    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:43:49.360501 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:43:49.360445    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" podStartSLOduration=1.6150203840000001 podStartE2EDuration="5.360429937s" totalImagesPullingTime="3.745409553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:49.35872214 +0000 UTC m=+433.188549373" watchObservedRunningTime="2026-09-10 23:43:49.360429937 +0000 UTC m=+433.190257159"
Sep 10 23:44:00.344955 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:00.344874    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 10 23:44:22.499329 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.499290    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:44:22.516483 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.516453    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:44:22.516651 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.516549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:44:22.518821 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.518752    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rcb9d\""
Sep 10 23:44:22.545173 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.545149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sh9n\" (UniqueName: \"kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n\") pod \"authorino-59c974bd48-lhjq7\" (UID: \"29ba5d41-a2f1-405b-bc30-06e1505da0c8\") " pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:44:22.646383 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.646355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sh9n\" (UniqueName: \"kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n\") pod \"authorino-59c974bd48-lhjq7\" (UID: \"29ba5d41-a2f1-405b-bc30-06e1505da0c8\") " pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:44:22.654289 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.654265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sh9n\" (UniqueName: \"kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n\") pod \"authorino-59c974bd48-lhjq7\" (UID: \"29ba5d41-a2f1-405b-bc30-06e1505da0c8\") " pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:44:22.825208 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:22.825126    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:44:23.091489 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:23.091461    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:44:23.094093 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:44:23.094068    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29ba5d41_a2f1_405b_bc30_06e1505da0c8.slice/crio-408acc92e178430cd6e53cda4b46fb2be2b0525010231c8d1753894b984d2606 WatchSource:0}: Error finding container 408acc92e178430cd6e53cda4b46fb2be2b0525010231c8d1753894b984d2606: Status 404 returned error can't find the container with id 408acc92e178430cd6e53cda4b46fb2be2b0525010231c8d1753894b984d2606
Sep 10 23:44:23.479365 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:23.479291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lhjq7" event={"ID":"29ba5d41-a2f1-405b-bc30-06e1505da0c8","Type":"ContainerStarted","Data":"408acc92e178430cd6e53cda4b46fb2be2b0525010231c8d1753894b984d2606"}
Sep 10 23:44:25.489497 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:25.489459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lhjq7" event={"ID":"29ba5d41-a2f1-405b-bc30-06e1505da0c8","Type":"ContainerStarted","Data":"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"}
Sep 10 23:44:25.507564 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:44:25.507512    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-lhjq7" podStartSLOduration=1.5904222780000001 podStartE2EDuration="3.507495136s" totalImagesPullingTime="1.917072858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:25.507151457 +0000 UTC m=+469.336978675" watchObservedRunningTime="2026-09-10 23:44:25.507495136 +0000 UTC m=+469.337322356"
Sep 10 23:45:10.023466 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.023434    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:45:10.023992 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.023601    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-lhjq7" podUID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" containerName="authorino" containerID="cri-o://1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20" gracePeriod=30
Sep 10 23:45:10.259286 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.259263    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:45:10.445988 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.445955    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sh9n\" (UniqueName: \"kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n\") pod \"29ba5d41-a2f1-405b-bc30-06e1505da0c8\" (UID: \"29ba5d41-a2f1-405b-bc30-06e1505da0c8\") "
Sep 10 23:45:10.448146 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.448116    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n" (OuterVolumeSpecName: "kube-api-access-5sh9n") pod "29ba5d41-a2f1-405b-bc30-06e1505da0c8" (UID: "29ba5d41-a2f1-405b-bc30-06e1505da0c8"). InnerVolumeSpecName "kube-api-access-5sh9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:45:10.546553 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.546520    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sh9n\" (UniqueName: \"kubernetes.io/projected/29ba5d41-a2f1-405b-bc30-06e1505da0c8-kube-api-access-5sh9n\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:45:10.675816 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.675785    2574 generic.go:356] "Generic (PLEG): container finished" podID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" containerID="1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20" exitCode=0
Sep 10 23:45:10.675974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.675830    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lhjq7"
Sep 10 23:45:10.675974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.675843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lhjq7" event={"ID":"29ba5d41-a2f1-405b-bc30-06e1505da0c8","Type":"ContainerDied","Data":"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"}
Sep 10 23:45:10.675974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.675874    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lhjq7" event={"ID":"29ba5d41-a2f1-405b-bc30-06e1505da0c8","Type":"ContainerDied","Data":"408acc92e178430cd6e53cda4b46fb2be2b0525010231c8d1753894b984d2606"}
Sep 10 23:45:10.675974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.675897    2574 scope.go:117] "RemoveContainer" containerID="1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"
Sep 10 23:45:10.683456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.683441    2574 scope.go:117] "RemoveContainer" containerID="1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"
Sep 10 23:45:10.683705 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:45:10.683687    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20\": container with ID starting with 1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20 not found: ID does not exist" containerID="1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"
Sep 10 23:45:10.683874 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.683717    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20"} err="failed to get container status \"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20\": rpc error: code = NotFound desc = could not find container \"1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20\": container with ID starting with 1429e64ecb5af1308dd430fe2b65924a2373a4823d24611bb2a3ff8669620c20 not found: ID does not exist"
Sep 10 23:45:10.690038 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.690015    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:45:10.693726 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:10.693694    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lhjq7"]
Sep 10 23:45:12.614112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:45:12.614078    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" path="/var/lib/kubelet/pods/29ba5d41-a2f1-405b-bc30-06e1505da0c8/volumes"
Sep 10 23:46:29.617716 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.617681    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"]
Sep 10 23:46:29.618131 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.618026    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" containerName="authorino"
Sep 10 23:46:29.618131 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.618038    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" containerName="authorino"
Sep 10 23:46:29.618131 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.618098    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ba5d41-a2f1-405b-bc30-06e1505da0c8" containerName="authorino"
Sep 10 23:46:29.621238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.621219    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.623452 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.623428    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:46:29.623562 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.623508    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:46:29.623989 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.623972    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 23:46:29.624089 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.623994    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rmfcp\""
Sep 10 23:46:29.630756 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.630732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"]
Sep 10 23:46:29.678342 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf58q\" (UniqueName: \"kubernetes.io/projected/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-kube-api-access-tf58q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.678480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.678480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.678480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.678635 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.678635 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.678557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf58q\" (UniqueName: \"kubernetes.io/projected/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-kube-api-access-tf58q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.779995 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.779963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.780105 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.780008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.780105 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.780038    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.781714 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.781695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.782073 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.782052    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.787831 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.787813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf58q\" (UniqueName: \"kubernetes.io/projected/4a38bf0d-b624-4da9-aea0-6eef11f0be6a-kube-api-access-tf58q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-c9v7k\" (UID: \"4a38bf0d-b624-4da9-aea0-6eef11f0be6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:29.932836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:29.932725    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:30.207079 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:30.207050    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"]
Sep 10 23:46:30.209386 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:46:30.209360    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a38bf0d_b624_4da9_aea0_6eef11f0be6a.slice/crio-ed97d497b708fa1e0ffe6556d94d50966104f42631ba44d2ee34176bdd9eeb98 WatchSource:0}: Error finding container ed97d497b708fa1e0ffe6556d94d50966104f42631ba44d2ee34176bdd9eeb98: Status 404 returned error can't find the container with id ed97d497b708fa1e0ffe6556d94d50966104f42631ba44d2ee34176bdd9eeb98
Sep 10 23:46:31.012918 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:31.012875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k" event={"ID":"4a38bf0d-b624-4da9-aea0-6eef11f0be6a","Type":"ContainerStarted","Data":"ed97d497b708fa1e0ffe6556d94d50966104f42631ba44d2ee34176bdd9eeb98"}
Sep 10 23:46:32.018861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:32.018754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k" event={"ID":"4a38bf0d-b624-4da9-aea0-6eef11f0be6a","Type":"ContainerStarted","Data":"b4cc4411ae656f488c688e5147d1028b0082e1e9690481c39c0d5d95695e6a8d"}
Sep 10 23:46:32.035363 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:32.035310    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k" podStartSLOduration=1.603617531 podStartE2EDuration="3.035290338s" totalImagesPullingTime="1.431672807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:32.034309328 +0000 UTC m=+595.864136562" watchObservedRunningTime="2026-09-10 23:46:32.035290338 +0000 UTC m=+595.865117556"
Sep 10 23:46:39.933587 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:39.933546    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:39.936004 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:39.933615    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:39.945966 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:39.945947    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:40.067462 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:40.067436    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-c9v7k"
Sep 10 23:46:54.823284 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.823242    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"]
Sep 10 23:46:54.827108 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.827089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.829207 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.829182    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 23:46:54.834544 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.834522    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"]
Sep 10 23:46:54.880445 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.880584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.880584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c56541-8428-49a6-b6ea-a3caa02bab04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.880584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.880707 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqk76\" (UniqueName: \"kubernetes.io/projected/b6c56541-8428-49a6-b6ea-a3caa02bab04-kube-api-access-wqk76\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.880707 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.880617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.981827 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.981792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqk76\" (UniqueName: \"kubernetes.io/projected/b6c56541-8428-49a6-b6ea-a3caa02bab04-kube-api-access-wqk76\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.981999 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.981833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.981999 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.981898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.981999 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.981966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.981999 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.981990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c56541-8428-49a6-b6ea-a3caa02bab04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.982173 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.982017    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.982298 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.982264    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.982414 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.982326    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.982414 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.982381    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.984309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.984286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c56541-8428-49a6-b6ea-a3caa02bab04-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.984512 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.984497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c56541-8428-49a6-b6ea-a3caa02bab04-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:54.989547 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:54.989524    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqk76\" (UniqueName: \"kubernetes.io/projected/b6c56541-8428-49a6-b6ea-a3caa02bab04-kube-api-access-wqk76\") pod \"e2e-trlp-test-simulated-kserve-598466548d-llgwd\" (UID: \"b6c56541-8428-49a6-b6ea-a3caa02bab04\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:55.138207 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:55.138175    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:46:55.405535 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:55.405429    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"]
Sep 10 23:46:55.408261 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:46:55.408233    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c56541_8428_49a6_b6ea_a3caa02bab04.slice/crio-b7955ae5dc8c7786e81abcd6fc1b0cd81ad7e2660856910dcd7c0925b2ba5c3c WatchSource:0}: Error finding container b7955ae5dc8c7786e81abcd6fc1b0cd81ad7e2660856910dcd7c0925b2ba5c3c: Status 404 returned error can't find the container with id b7955ae5dc8c7786e81abcd6fc1b0cd81ad7e2660856910dcd7c0925b2ba5c3c
Sep 10 23:46:55.409973 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:55.409954    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:46:56.137145 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:56.137108    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd" event={"ID":"b6c56541-8428-49a6-b6ea-a3caa02bab04","Type":"ContainerStarted","Data":"34c342cef1fdbdd01d1251d81dfe939df99fa4ca9580be061022e41f38bc7762"}
Sep 10 23:46:56.137526 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:56.137163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd" event={"ID":"b6c56541-8428-49a6-b6ea-a3caa02bab04","Type":"ContainerStarted","Data":"b7955ae5dc8c7786e81abcd6fc1b0cd81ad7e2660856910dcd7c0925b2ba5c3c"}
Sep 10 23:46:56.153470 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:46:56.153413    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd" podStartSLOduration=1.9232743079999999 podStartE2EDuration="2.153397719s" totalImagesPullingTime="230.123411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:56.152356073 +0000 UTC m=+619.982183316" watchObservedRunningTime="2026-09-10 23:46:56.153397719 +0000 UTC m=+619.983224938"
Sep 10 23:47:05.138794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:05.138692    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:47:05.138794 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:05.138787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:47:05.150699 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:05.150676    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:47:05.186035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:05.186016    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-llgwd"
Sep 10 23:47:29.815891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.815854    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"]
Sep 10 23:47:29.819808 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.819790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.821852 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.821829    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 23:47:29.826836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.826813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"]
Sep 10 23:47:29.880459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.880618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98s2\" (UniqueName: \"kubernetes.io/projected/580369c3-9510-4350-a7b2-5fd43897a6c7-kube-api-access-h98s2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.880618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580369c3-9510-4350-a7b2-5fd43897a6c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.880618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.880618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.880753 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.880659    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981754 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98s2\" (UniqueName: \"kubernetes.io/projected/580369c3-9510-4350-a7b2-5fd43897a6c7-kube-api-access-h98s2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580369c3-9510-4350-a7b2-5fd43897a6c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.981968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.981965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.982238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.982196    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.982291 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.982235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.982344 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.982308    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.984348 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.984322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580369c3-9510-4350-a7b2-5fd43897a6c7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.984486 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.984469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580369c3-9510-4350-a7b2-5fd43897a6c7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:29.989965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:29.989940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98s2\" (UniqueName: \"kubernetes.io/projected/580369c3-9510-4350-a7b2-5fd43897a6c7-kube-api-access-h98s2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pw47c\" (UID: \"580369c3-9510-4350-a7b2-5fd43897a6c7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:30.133984 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:30.133947    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:30.403384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:30.403301    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"]
Sep 10 23:47:30.405485 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:47:30.405457    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580369c3_9510_4350_a7b2_5fd43897a6c7.slice/crio-d0867afd9ab97ed4c0389a6b3970f9a38c847c087fc81a0435539faf399a294f WatchSource:0}: Error finding container d0867afd9ab97ed4c0389a6b3970f9a38c847c087fc81a0435539faf399a294f: Status 404 returned error can't find the container with id d0867afd9ab97ed4c0389a6b3970f9a38c847c087fc81a0435539faf399a294f
Sep 10 23:47:31.290110 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:31.290074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c" event={"ID":"580369c3-9510-4350-a7b2-5fd43897a6c7","Type":"ContainerStarted","Data":"9072a6637fb15af841f79d0e9192fbb87d9d9fdf9cf830c96cd18eddec784e4d"}
Sep 10 23:47:31.290472 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:31.290127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c" event={"ID":"580369c3-9510-4350-a7b2-5fd43897a6c7","Type":"ContainerStarted","Data":"d0867afd9ab97ed4c0389a6b3970f9a38c847c087fc81a0435539faf399a294f"}
Sep 10 23:47:31.306492 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:31.306441    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c" podStartSLOduration=2.119905087 podStartE2EDuration="2.306426461s" totalImagesPullingTime="186.521374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:31.304552187 +0000 UTC m=+655.134379417" watchObservedRunningTime="2026-09-10 23:47:31.306426461 +0000 UTC m=+655.136253679"
Sep 10 23:47:40.134235 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:40.134198    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:40.134637 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:40.134274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:40.146957 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:40.146932    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:40.339484 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:40.339457    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pw47c"
Sep 10 23:47:50.731109 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.731066    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"]
Sep 10 23:47:50.734446 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.734426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.736480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.736463    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 23:47:50.742319 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.742294    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"]
Sep 10 23:47:50.873761 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.873940 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.873940 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.874020 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.874020 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppgj\" (UniqueName: \"kubernetes.io/projected/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-kube-api-access-5ppgj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.874020 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.873997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974603 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974603 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppgj\" (UniqueName: \"kubernetes.io/projected/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-kube-api-access-5ppgj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974603 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974756 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974756 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974899 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974880    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.974981 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.974961    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.975136 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.975108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.976924 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.976897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.977160 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.977144    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:50.981416 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:50.981364    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppgj\" (UniqueName: \"kubernetes.io/projected/6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437-kube-api-access-5ppgj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8\" (UID: \"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:51.044523 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:51.044493    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:47:51.312851 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:51.312732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"]
Sep 10 23:47:51.315632 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:47:51.315604    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a4dd1a8_ff15_4edb_8a2b_381b1b8f0437.slice/crio-f527d16d429850c68e2221d4a5ebe64142db3f0bc28167bbe98d82417b179728 WatchSource:0}: Error finding container f527d16d429850c68e2221d4a5ebe64142db3f0bc28167bbe98d82417b179728: Status 404 returned error can't find the container with id f527d16d429850c68e2221d4a5ebe64142db3f0bc28167bbe98d82417b179728
Sep 10 23:47:51.374688 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:51.374659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8" event={"ID":"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437","Type":"ContainerStarted","Data":"f527d16d429850c68e2221d4a5ebe64142db3f0bc28167bbe98d82417b179728"}
Sep 10 23:47:52.381387 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:52.381355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8" event={"ID":"6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437","Type":"ContainerStarted","Data":"540f5ce51a4b7a0cbe940767f7c282abf076d88b7ed74d1358c1b6470419b6ce"}
Sep 10 23:47:52.399703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:47:52.399659    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8" podStartSLOduration=2.196768562 podStartE2EDuration="2.39964284s" totalImagesPullingTime="202.874278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:52.397891882 +0000 UTC m=+676.227719103" watchObservedRunningTime="2026-09-10 23:47:52.39964284 +0000 UTC m=+676.229470058"
Sep 10 23:48:01.045307 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:48:01.045268    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:48:01.045841 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:48:01.045397    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:48:01.057533 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:48:01.057510    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:48:01.431460 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:48:01.431431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8"
Sep 10 23:49:01.072613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.072581    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bdcb74d9-g99n4"]
Sep 10 23:49:01.076094 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.076076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.078823 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.078803    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rcb9d\""
Sep 10 23:49:01.078962 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.078802    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:49:01.082663 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.082639    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bdcb74d9-g99n4"]
Sep 10 23:49:01.169651 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.169614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9249f220-6368-4389-98ce-b17ccecfc028-tls-cert\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.169834 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.169711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdvf\" (UniqueName: \"kubernetes.io/projected/9249f220-6368-4389-98ce-b17ccecfc028-kube-api-access-dfdvf\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.270211 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.270168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdvf\" (UniqueName: \"kubernetes.io/projected/9249f220-6368-4389-98ce-b17ccecfc028-kube-api-access-dfdvf\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.270381 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.270259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9249f220-6368-4389-98ce-b17ccecfc028-tls-cert\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.272689 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.272659    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9249f220-6368-4389-98ce-b17ccecfc028-tls-cert\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.277990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.277969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdvf\" (UniqueName: \"kubernetes.io/projected/9249f220-6368-4389-98ce-b17ccecfc028-kube-api-access-dfdvf\") pod \"authorino-bdcb74d9-g99n4\" (UID: \"9249f220-6368-4389-98ce-b17ccecfc028\") " pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.386731 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.386700    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bdcb74d9-g99n4"
Sep 10 23:49:01.654281 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.654252    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bdcb74d9-g99n4"]
Sep 10 23:49:01.656861 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:49:01.656832    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9249f220_6368_4389_98ce_b17ccecfc028.slice/crio-34c3c754cbbe46e3bf840664fe968fd6f33e9271c735fef7eb72812b4042a185 WatchSource:0}: Error finding container 34c3c754cbbe46e3bf840664fe968fd6f33e9271c735fef7eb72812b4042a185: Status 404 returned error can't find the container with id 34c3c754cbbe46e3bf840664fe968fd6f33e9271c735fef7eb72812b4042a185
Sep 10 23:49:01.685325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:01.685292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bdcb74d9-g99n4" event={"ID":"9249f220-6368-4389-98ce-b17ccecfc028","Type":"ContainerStarted","Data":"34c3c754cbbe46e3bf840664fe968fd6f33e9271c735fef7eb72812b4042a185"}
Sep 10 23:49:02.691742 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:02.691694    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bdcb74d9-g99n4" event={"ID":"9249f220-6368-4389-98ce-b17ccecfc028","Type":"ContainerStarted","Data":"230d51aa6a104d63107672fc493f06fd5bb07daed4ee5a0174b8da0b7cefb637"}
Sep 10 23:49:02.712996 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:02.712933    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bdcb74d9-g99n4" podStartSLOduration=0.902704217 podStartE2EDuration="1.71289087s" totalImagesPullingTime="810.186653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:02.712157223 +0000 UTC m=+746.541984493" watchObservedRunningTime="2026-09-10 23:49:02.71289087 +0000 UTC m=+746.542718096"
Sep 10 23:49:32.995236 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:32.995198    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:49:32.998269 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:32.998251    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.000799 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.000760    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e027297d\""
Sep 10 23:49:33.000929 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.000909    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e027297d-dockercfg-cjwv2\""
Sep 10 23:49:33.017990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.017966    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:49:33.152292 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.152259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.152436 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.152347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxr6s\" (UniqueName: \"kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.253406 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.253333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.253406 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.253397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxr6s\" (UniqueName: \"kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.259798 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.254649    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.272949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.272915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxr6s\" (UniqueName: \"kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s\") pod \"payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") " pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.308674 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.308636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:33.605258 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.605225    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:49:33.608204 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:49:33.608178    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e6da58_6347_4d36_874b_dc6c59ade3e6.slice/crio-2813889559e81529eb9e6ffa4a89436a2a7e0ca271da48883601c0c71eb7126c WatchSource:0}: Error finding container 2813889559e81529eb9e6ffa4a89436a2a7e0ca271da48883601c0c71eb7126c: Status 404 returned error can't find the container with id 2813889559e81529eb9e6ffa4a89436a2a7e0ca271da48883601c0c71eb7126c
Sep 10 23:49:33.830385 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:33.830352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" event={"ID":"14e6da58-6347-4d36-874b-dc6c59ade3e6","Type":"ContainerStarted","Data":"2813889559e81529eb9e6ffa4a89436a2a7e0ca271da48883601c0c71eb7126c"}
Sep 10 23:49:36.851096 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:36.851057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" event={"ID":"14e6da58-6347-4d36-874b-dc6c59ade3e6","Type":"ContainerStarted","Data":"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"}
Sep 10 23:49:36.851443 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:36.851181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:36.891952 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:36.891888    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" podStartSLOduration=2.571505848 podStartE2EDuration="4.89187154s" totalImagesPullingTime="2.320365692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:36.889305785 +0000 UTC m=+780.719133003" watchObservedRunningTime="2026-09-10 23:49:36.89187154 +0000 UTC m=+780.721698758"
Sep 10 23:49:47.856940 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:47.856904    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:49:55.609325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.609289    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:49:55.614607 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.614584    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.618212 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.618190    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7f598e-dockercfg-d8mrr\""
Sep 10 23:49:55.618474 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.618460    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7f598e\""
Sep 10 23:49:55.629576 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.629552    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:49:55.636798 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.636751    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.636875 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.636829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzdm\" (UniqueName: \"kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.656709 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.656683    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:49:55.659641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.659619    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.670909 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.670880    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:49:55.737486 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.737451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzdm\" (UniqueName: \"kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.737657 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.737515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrdw\" (UniqueName: \"kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.737730 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.737660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.737811 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.737744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.738426 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.738404    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.748212 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.748185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzdm\" (UniqueName: \"kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm\") pod \"payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.838989 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.838959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.839113 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.839060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrdw\" (UniqueName: \"kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.839694 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.839676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.847600 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.847575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrdw\" (UniqueName: \"kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw\") pod \"payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:55.923993 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.923931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:55.969568 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:55.969536    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:56.163936 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.163907    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"]
Sep 10 23:49:56.167664 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.167647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:49:56.169977 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.169950    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:49:56.169977 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.169952    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:49:56.175699 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.175534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"]
Sep 10 23:49:56.195882 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.195808    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:49:56.197907 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:49:56.197883    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b9246a_ba44_427c_95f9_580dd96dfc47.slice/crio-6f872f704c889b3052ed98b7353b8093e4b6911de228861fcc6bca1cbc4238cb WatchSource:0}: Error finding container 6f872f704c889b3052ed98b7353b8093e4b6911de228861fcc6bca1cbc4238cb: Status 404 returned error can't find the container with id 6f872f704c889b3052ed98b7353b8093e4b6911de228861fcc6bca1cbc4238cb
Sep 10 23:49:56.242161 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.242138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e027297d-s28bc\" (UID: \"25816e53-8b52-4aa5-b54e-6cca6a4a2cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:49:56.253112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.253089    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:49:56.256031 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:49:56.256009    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd245709a_13fd_4882_a3b7_ad456916fcde.slice/crio-14faedd4cca7871771fb1d0f11a878d96498096fe7562471a3fae059688bbeb3 WatchSource:0}: Error finding container 14faedd4cca7871771fb1d0f11a878d96498096fe7562471a3fae059688bbeb3: Status 404 returned error can't find the container with id 14faedd4cca7871771fb1d0f11a878d96498096fe7562471a3fae059688bbeb3
Sep 10 23:49:56.343218 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.343187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e027297d-s28bc\" (UID: \"25816e53-8b52-4aa5-b54e-6cca6a4a2cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:49:56.343800 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.343753    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e027297d-s28bc\" (UID: \"25816e53-8b52-4aa5-b54e-6cca6a4a2cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:49:56.479643 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.479562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:49:56.759434 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.759407    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"]
Sep 10 23:49:56.762440 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:49:56.762406    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25816e53_8b52_4aa5_b54e_6cca6a4a2cce.slice/crio-dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464 WatchSource:0}: Error finding container dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464: Status 404 returned error can't find the container with id dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464
Sep 10 23:49:56.949950 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.949895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" event={"ID":"d1b9246a-ba44-427c-95f9-580dd96dfc47","Type":"ContainerStarted","Data":"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"}
Sep 10 23:49:56.950134 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.949977    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:49:56.950134 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.950004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" event={"ID":"d1b9246a-ba44-427c-95f9-580dd96dfc47","Type":"ContainerStarted","Data":"6f872f704c889b3052ed98b7353b8093e4b6911de228861fcc6bca1cbc4238cb"}
Sep 10 23:49:56.951703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.951669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" event={"ID":"d245709a-13fd-4882-a3b7-ad456916fcde","Type":"ContainerStarted","Data":"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"}
Sep 10 23:49:56.951860 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.951728    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" event={"ID":"d245709a-13fd-4882-a3b7-ad456916fcde","Type":"ContainerStarted","Data":"14faedd4cca7871771fb1d0f11a878d96498096fe7562471a3fae059688bbeb3"}
Sep 10 23:49:56.951860 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.951759    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:49:56.952916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.952892    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc" event={"ID":"25816e53-8b52-4aa5-b54e-6cca6a4a2cce","Type":"ContainerStarted","Data":"dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464"}
Sep 10 23:49:56.970235 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.970194    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" podStartSLOduration=1.970181787 podStartE2EDuration="1.970181787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:56.967306305 +0000 UTC m=+800.797133525" watchObservedRunningTime="2026-09-10 23:49:56.970181787 +0000 UTC m=+800.800009006"
Sep 10 23:49:56.990135 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:56.990086    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" podStartSLOduration=1.990067847 podStartE2EDuration="1.990067847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:56.986608972 +0000 UTC m=+800.816436192" watchObservedRunningTime="2026-09-10 23:49:56.990067847 +0000 UTC m=+800.819895067"
Sep 10 23:49:59.971065 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:59.971025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc" event={"ID":"25816e53-8b52-4aa5-b54e-6cca6a4a2cce","Type":"ContainerStarted","Data":"6abeea0d89f1dbe2522e54d1ba6a7a9f758b305eddcb0c231871db12458db2bc"}
Sep 10 23:49:59.989032 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:49:59.988977    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc" podStartSLOduration=1.085696695 podStartE2EDuration="3.988957959s" totalImagesPullingTime="2.903261264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:59.988481247 +0000 UTC m=+803.818308490" watchObservedRunningTime="2026-09-10 23:49:59.988957959 +0000 UTC m=+803.818785180"
Sep 10 23:50:00.977473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:00.977437    2574 generic.go:356] "Generic (PLEG): container finished" podID="25816e53-8b52-4aa5-b54e-6cca6a4a2cce" containerID="6abeea0d89f1dbe2522e54d1ba6a7a9f758b305eddcb0c231871db12458db2bc" exitCode=0
Sep 10 23:50:00.977936 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:00.977523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc" event={"ID":"25816e53-8b52-4aa5-b54e-6cca6a4a2cce","Type":"ContainerDied","Data":"6abeea0d89f1dbe2522e54d1ba6a7a9f758b305eddcb0c231871db12458db2bc"}
Sep 10 23:50:02.112339 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.112311    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:50:02.193879 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.193847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca\") pod \"25816e53-8b52-4aa5-b54e-6cca6a4a2cce\" (UID: \"25816e53-8b52-4aa5-b54e-6cca6a4a2cce\") "
Sep 10 23:50:02.194246 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.194223    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25816e53-8b52-4aa5-b54e-6cca6a4a2cce" (UID: "25816e53-8b52-4aa5-b54e-6cca6a4a2cce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:02.295049 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.294963    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25816e53-8b52-4aa5-b54e-6cca6a4a2cce-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:02.987974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.987902    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"
Sep 10 23:50:02.988111 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.987900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc" event={"ID":"25816e53-8b52-4aa5-b54e-6cca6a4a2cce","Type":"ContainerDied","Data":"dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464"}
Sep 10 23:50:02.988111 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:02.988034    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dffd03ee2c83de4fcea0a481866dfe575465272896cfd1f0d249958520a77464"
Sep 10 23:50:07.959874 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:07.959839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:50:07.960407 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:07.959908    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:50:12.730122 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:12.730088    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:50:12.730613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:12.730329    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" podUID="14e6da58-6347-4d36-874b-dc6c59ade3e6" containerName="payload-processing" containerID="cri-o://dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4" gracePeriod=30
Sep 10 23:50:12.921606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:12.921571    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"]
Sep 10 23:50:12.926347 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:12.926321    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e027297d-s28bc"]
Sep 10 23:50:12.982405 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:12.982341    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:50:13.036272 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.036235    2574 generic.go:356] "Generic (PLEG): container finished" podID="14e6da58-6347-4d36-874b-dc6c59ade3e6" containerID="dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4" exitCode=0
Sep 10 23:50:13.036272 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.036272    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"
Sep 10 23:50:13.036697 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.036278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" event={"ID":"14e6da58-6347-4d36-874b-dc6c59ade3e6","Type":"ContainerDied","Data":"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"}
Sep 10 23:50:13.036697 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.036334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv" event={"ID":"14e6da58-6347-4d36-874b-dc6c59ade3e6","Type":"ContainerDied","Data":"2813889559e81529eb9e6ffa4a89436a2a7e0ca271da48883601c0c71eb7126c"}
Sep 10 23:50:13.036697 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.036369    2574 scope.go:117] "RemoveContainer" containerID="dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"
Sep 10 23:50:13.046624 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.046603    2574 scope.go:117] "RemoveContainer" containerID="dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"
Sep 10 23:50:13.046952 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:50:13.046933    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4\": container with ID starting with dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4 not found: ID does not exist" containerID="dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"
Sep 10 23:50:13.047036 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.046961    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4"} err="failed to get container status \"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4\": rpc error: code = NotFound desc = could not find container \"dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4\": container with ID starting with dcde8939b26fcf9b06ea3632d961cdad14916cb606814495610cedeb7c4af7b4 not found: ID does not exist"
Sep 10 23:50:13.094400 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.094375    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxr6s\" (UniqueName: \"kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s\") pod \"14e6da58-6347-4d36-874b-dc6c59ade3e6\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") "
Sep 10 23:50:13.094512 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.094500    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume\") pod \"14e6da58-6347-4d36-874b-dc6c59ade3e6\" (UID: \"14e6da58-6347-4d36-874b-dc6c59ade3e6\") "
Sep 10 23:50:13.094847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.094824    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14e6da58-6347-4d36-874b-dc6c59ade3e6" (UID: "14e6da58-6347-4d36-874b-dc6c59ade3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:13.096359 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.096339    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s" (OuterVolumeSpecName: "kube-api-access-dxr6s") pod "14e6da58-6347-4d36-874b-dc6c59ade3e6" (UID: "14e6da58-6347-4d36-874b-dc6c59ade3e6"). InnerVolumeSpecName "kube-api-access-dxr6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:13.195705 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.195674    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e6da58-6347-4d36-874b-dc6c59ade3e6-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.195705 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.195699    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxr6s\" (UniqueName: \"kubernetes.io/projected/14e6da58-6347-4d36-874b-dc6c59ade3e6-kube-api-access-dxr6s\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:13.357427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.357397    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:50:13.365782 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:13.365747    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e027297d-6b8fdb4487-jvjcv"]
Sep 10 23:50:14.614384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:14.614350    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e6da58-6347-4d36-874b-dc6c59ade3e6" path="/var/lib/kubelet/pods/14e6da58-6347-4d36-874b-dc6c59ade3e6/volumes"
Sep 10 23:50:14.614719 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:14.614676    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25816e53-8b52-4aa5-b54e-6cca6a4a2cce" path="/var/lib/kubelet/pods/25816e53-8b52-4aa5-b54e-6cca6a4a2cce/volumes"
Sep 10 23:50:18.663722 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.663686    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:18.664119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664028    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25816e53-8b52-4aa5-b54e-6cca6a4a2cce" containerName="revoke-keys"
Sep 10 23:50:18.664119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664040    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25816e53-8b52-4aa5-b54e-6cca6a4a2cce" containerName="revoke-keys"
Sep 10 23:50:18.664119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664055    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e6da58-6347-4d36-874b-dc6c59ade3e6" containerName="payload-processing"
Sep 10 23:50:18.664119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664061    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e6da58-6347-4d36-874b-dc6c59ade3e6" containerName="payload-processing"
Sep 10 23:50:18.664119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664119    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e6da58-6347-4d36-874b-dc6c59ade3e6" containerName="payload-processing"
Sep 10 23:50:18.664272 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.664129    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="25816e53-8b52-4aa5-b54e-6cca6a4a2cce" containerName="revoke-keys"
Sep 10 23:50:18.668386 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.668370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.671963 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.671938    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6a8995da\""
Sep 10 23:50:18.672086 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.671948    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6a8995da-dockercfg-pw5xh\""
Sep 10 23:50:18.673582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.673561    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:18.676982 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.676963    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.683429 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.683397    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:18.689246 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.689221    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:18.745477 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.745443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.745633 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.745486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkj8\" (UniqueName: \"kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.745633 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.745514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8dn\" (UniqueName: \"kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.745710 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.745637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.847043 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.846995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.847228 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.847151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.847228 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.847172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkj8\" (UniqueName: \"kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.847228 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.847194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8dn\" (UniqueName: \"kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.847792 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.847750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.847840 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.847827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.856230 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.856201    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkj8\" (UniqueName: \"kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8\") pod \"payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:18.856322 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.856273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8dn\" (UniqueName: \"kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn\") pod \"payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") " pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.978211 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.978117    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:18.989919 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:18.989792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:19.260214 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:19.260192    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:19.262840 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:19.262801    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b1392c_9ac6_4377_b62d_22e3a14d90b8.slice/crio-3a48ff1fc3c2d8de585694b05b3f55edf1cec4cf46bdd5afc9dbdb57f9352ac2 WatchSource:0}: Error finding container 3a48ff1fc3c2d8de585694b05b3f55edf1cec4cf46bdd5afc9dbdb57f9352ac2: Status 404 returned error can't find the container with id 3a48ff1fc3c2d8de585694b05b3f55edf1cec4cf46bdd5afc9dbdb57f9352ac2
Sep 10 23:50:19.276397 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:19.276261    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:19.278446 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:19.278352    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06af46c_8fcd_480a_965e_aaf3f36acff9.slice/crio-cf6b65d7ded0679d30423e7b3b6ef4bf43376ab19290b08fd0a8b8b26ff76b3f WatchSource:0}: Error finding container cf6b65d7ded0679d30423e7b3b6ef4bf43376ab19290b08fd0a8b8b26ff76b3f: Status 404 returned error can't find the container with id cf6b65d7ded0679d30423e7b3b6ef4bf43376ab19290b08fd0a8b8b26ff76b3f
Sep 10 23:50:20.076130 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.076091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" event={"ID":"25b1392c-9ac6-4377-b62d-22e3a14d90b8","Type":"ContainerStarted","Data":"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"}
Sep 10 23:50:20.076577 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.076158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" event={"ID":"25b1392c-9ac6-4377-b62d-22e3a14d90b8","Type":"ContainerStarted","Data":"3a48ff1fc3c2d8de585694b05b3f55edf1cec4cf46bdd5afc9dbdb57f9352ac2"}
Sep 10 23:50:20.076577 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.076187    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:20.077834 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.077807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" event={"ID":"b06af46c-8fcd-480a-965e-aaf3f36acff9","Type":"ContainerStarted","Data":"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"}
Sep 10 23:50:20.077961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.077866    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:20.077961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.077896    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" event={"ID":"b06af46c-8fcd-480a-965e-aaf3f36acff9","Type":"ContainerStarted","Data":"cf6b65d7ded0679d30423e7b3b6ef4bf43376ab19290b08fd0a8b8b26ff76b3f"}
Sep 10 23:50:20.093282 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.093221    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" podStartSLOduration=2.093204561 podStartE2EDuration="2.093204561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:20.092352718 +0000 UTC m=+823.922179938" watchObservedRunningTime="2026-09-10 23:50:20.093204561 +0000 UTC m=+823.923031785"
Sep 10 23:50:20.112500 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:20.112452    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" podStartSLOduration=2.112434353 podStartE2EDuration="2.112434353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:20.110353474 +0000 UTC m=+823.940180700" watchObservedRunningTime="2026-09-10 23:50:20.112434353 +0000 UTC m=+823.942261622"
Sep 10 23:50:31.084229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:31.084192    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:31.084638 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:31.084270    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:39.635524 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.635487    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:50:39.640236 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.640215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.643005 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.642978    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-74f1c1\"/\"default-dockercfg-fhpfw\""
Sep 10 23:50:39.643133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.643015    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-74f1c1\"/\"openshift-service-ca.crt\""
Sep 10 23:50:39.643133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.643093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-74f1c1\"/\"autores-model-74f1c1-kserve-self-signed-certs\""
Sep 10 23:50:39.644035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.644020    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-74f1c1\"/\"kube-root-ca.crt\""
Sep 10 23:50:39.648841 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.648817    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:50:39.738566 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5ln\" (UniqueName: \"kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.738721 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.738721 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.738847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.738847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.738948 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.738867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840372 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840372 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5ln\" (UniqueName: \"kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840372 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840372 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840580 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840580 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.840689 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.840651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.842480 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.842458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.842758 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.842742    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.848058 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.848035    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5ln\" (UniqueName: \"kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln\") pod \"autores-model-74f1c1-kserve-5bd6f4767-cqsg8\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") " pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:39.949817 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:39.949737    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:40.219999 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:40.219975    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:50:40.222155 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:40.222123    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71b6652b_94eb_488a_ab83_d083bccf5301.slice/crio-49d39c977c873345ecdd725dc5fcbb5139a5e7c217052dc628b39647698965f5 WatchSource:0}: Error finding container 49d39c977c873345ecdd725dc5fcbb5139a5e7c217052dc628b39647698965f5: Status 404 returned error can't find the container with id 49d39c977c873345ecdd725dc5fcbb5139a5e7c217052dc628b39647698965f5
Sep 10 23:50:41.174448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:41.174411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" event={"ID":"71b6652b-94eb-488a-ab83-d083bccf5301","Type":"ContainerStarted","Data":"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"}
Sep 10 23:50:41.174448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:41.174453    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" event={"ID":"71b6652b-94eb-488a-ab83-d083bccf5301","Type":"ContainerStarted","Data":"49d39c977c873345ecdd725dc5fcbb5139a5e7c217052dc628b39647698965f5"}
Sep 10 23:50:41.191166 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:41.191121    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" podStartSLOduration=2.191106719 podStartE2EDuration="2.191106719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:41.189489631 +0000 UTC m=+845.019316850" watchObservedRunningTime="2026-09-10 23:50:41.191106719 +0000 UTC m=+845.020933934"
Sep 10 23:50:42.287916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.287884    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"]
Sep 10 23:50:42.291295 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.291278    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:42.294937 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.294917    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:50:42.295112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.295094    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:50:42.311149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.306521    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"]
Sep 10 23:50:42.361385 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.361354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2\" (UID: \"123e15f8-23fd-4e6a-a004-6bd3e9616517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:42.462289 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.462262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2\" (UID: \"123e15f8-23fd-4e6a-a004-6bd3e9616517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:42.462906 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.462881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2\" (UID: \"123e15f8-23fd-4e6a-a004-6bd3e9616517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:42.601551 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.601519    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:42.875398 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:42.875370    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"]
Sep 10 23:50:42.877592 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:42.877567    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123e15f8_23fd_4e6a_a004_6bd3e9616517.slice/crio-d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589 WatchSource:0}: Error finding container d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589: Status 404 returned error can't find the container with id d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589
Sep 10 23:50:43.185780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:43.185681    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2" event={"ID":"123e15f8-23fd-4e6a-a004-6bd3e9616517","Type":"ContainerStarted","Data":"462aac5461dbc3b873906a223584131302a37b42829dd56b18ead9ff99a097c8"}
Sep 10 23:50:43.185780 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:43.185738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2" event={"ID":"123e15f8-23fd-4e6a-a004-6bd3e9616517","Type":"ContainerStarted","Data":"d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589"}
Sep 10 23:50:43.221793 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:43.221732    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2" podStartSLOduration=1.221714798 podStartE2EDuration="1.221714798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:43.219699863 +0000 UTC m=+847.049527093" watchObservedRunningTime="2026-09-10 23:50:43.221714798 +0000 UTC m=+847.051542017"
Sep 10 23:50:44.191520 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:44.191488    2574 generic.go:356] "Generic (PLEG): container finished" podID="123e15f8-23fd-4e6a-a004-6bd3e9616517" containerID="462aac5461dbc3b873906a223584131302a37b42829dd56b18ead9ff99a097c8" exitCode=0
Sep 10 23:50:44.191901 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:44.191570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2" event={"ID":"123e15f8-23fd-4e6a-a004-6bd3e9616517","Type":"ContainerDied","Data":"462aac5461dbc3b873906a223584131302a37b42829dd56b18ead9ff99a097c8"}
Sep 10 23:50:45.340332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:45.340307    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:45.386306 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:45.386276    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca\") pod \"123e15f8-23fd-4e6a-a004-6bd3e9616517\" (UID: \"123e15f8-23fd-4e6a-a004-6bd3e9616517\") "
Sep 10 23:50:45.386601 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:45.386579    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "123e15f8-23fd-4e6a-a004-6bd3e9616517" (UID: "123e15f8-23fd-4e6a-a004-6bd3e9616517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:45.487966 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:45.487905    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/123e15f8-23fd-4e6a-a004-6bd3e9616517-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:46.201841 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:46.201809    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"
Sep 10 23:50:46.202005 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:46.201812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2" event={"ID":"123e15f8-23fd-4e6a-a004-6bd3e9616517","Type":"ContainerDied","Data":"d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589"}
Sep 10 23:50:46.202005 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:46.201959    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d013e98289f467be191a901e1bbffbde6a606225bf4bedc98b3e178f1fab6589"
Sep 10 23:50:48.639714 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.639679    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:48.640203 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.639956    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" podUID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" containerName="payload-processing" containerID="cri-o://baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9" gracePeriod=30
Sep 10 23:50:48.653002 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.652973    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:48.653204 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.653179    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" podUID="b06af46c-8fcd-480a-965e-aaf3f36acff9" containerName="payload-pre-processing" containerID="cri-o://37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0" gracePeriod=30
Sep 10 23:50:48.926528 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.926491    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:48.932343 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.932317    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"]
Sep 10 23:50:48.938537 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.938481    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6a8995da-56tk2"]
Sep 10 23:50:48.957379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:48.957357    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:49.018362 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018322    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume\") pod \"b06af46c-8fcd-480a-965e-aaf3f36acff9\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") "
Sep 10 23:50:49.018362 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018372    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hkj8\" (UniqueName: \"kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8\") pod \"b06af46c-8fcd-480a-965e-aaf3f36acff9\" (UID: \"b06af46c-8fcd-480a-965e-aaf3f36acff9\") "
Sep 10 23:50:49.018581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018431    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume\") pod \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") "
Sep 10 23:50:49.018581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs8dn\" (UniqueName: \"kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn\") pod \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\" (UID: \"25b1392c-9ac6-4377-b62d-22e3a14d90b8\") "
Sep 10 23:50:49.018776 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018734    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b06af46c-8fcd-480a-965e-aaf3f36acff9" (UID: "b06af46c-8fcd-480a-965e-aaf3f36acff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:49.018865 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.018753    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25b1392c-9ac6-4377-b62d-22e3a14d90b8" (UID: "25b1392c-9ac6-4377-b62d-22e3a14d90b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:50:49.020660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.020635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn" (OuterVolumeSpecName: "kube-api-access-fs8dn") pod "25b1392c-9ac6-4377-b62d-22e3a14d90b8" (UID: "25b1392c-9ac6-4377-b62d-22e3a14d90b8"). InnerVolumeSpecName "kube-api-access-fs8dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:49.020660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.020649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8" (OuterVolumeSpecName: "kube-api-access-5hkj8") pod "b06af46c-8fcd-480a-965e-aaf3f36acff9" (UID: "b06af46c-8fcd-480a-965e-aaf3f36acff9"). InnerVolumeSpecName "kube-api-access-5hkj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:50:49.119322 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.119286    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06af46c-8fcd-480a-965e-aaf3f36acff9-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:49.119322 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.119313    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hkj8\" (UniqueName: \"kubernetes.io/projected/b06af46c-8fcd-480a-965e-aaf3f36acff9-kube-api-access-5hkj8\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:49.119322 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.119326    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1392c-9ac6-4377-b62d-22e3a14d90b8-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:49.119541 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.119338    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs8dn\" (UniqueName: \"kubernetes.io/projected/25b1392c-9ac6-4377-b62d-22e3a14d90b8-kube-api-access-fs8dn\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:50:49.217748 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.217663    2574 generic.go:356] "Generic (PLEG): container finished" podID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" containerID="baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9" exitCode=0
Sep 10 23:50:49.217748 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.217711    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"
Sep 10 23:50:49.217968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.217743    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" event={"ID":"25b1392c-9ac6-4377-b62d-22e3a14d90b8","Type":"ContainerDied","Data":"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"}
Sep 10 23:50:49.217968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.217824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92" event={"ID":"25b1392c-9ac6-4377-b62d-22e3a14d90b8","Type":"ContainerDied","Data":"3a48ff1fc3c2d8de585694b05b3f55edf1cec4cf46bdd5afc9dbdb57f9352ac2"}
Sep 10 23:50:49.217968 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.217853    2574 scope.go:117] "RemoveContainer" containerID="baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"
Sep 10 23:50:49.219337 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.219310    2574 generic.go:356] "Generic (PLEG): container finished" podID="b06af46c-8fcd-480a-965e-aaf3f36acff9" containerID="37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0" exitCode=0
Sep 10 23:50:49.219455 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.219350    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"
Sep 10 23:50:49.219455 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.219383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" event={"ID":"b06af46c-8fcd-480a-965e-aaf3f36acff9","Type":"ContainerDied","Data":"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"}
Sep 10 23:50:49.219455 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.219434    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr" event={"ID":"b06af46c-8fcd-480a-965e-aaf3f36acff9","Type":"ContainerDied","Data":"cf6b65d7ded0679d30423e7b3b6ef4bf43376ab19290b08fd0a8b8b26ff76b3f"}
Sep 10 23:50:49.228803 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.228787    2574 scope.go:117] "RemoveContainer" containerID="baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"
Sep 10 23:50:49.229123 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:50:49.229100    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9\": container with ID starting with baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9 not found: ID does not exist" containerID="baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"
Sep 10 23:50:49.229217 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.229134    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9"} err="failed to get container status \"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9\": rpc error: code = NotFound desc = could not find container \"baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9\": container with ID starting with baa379eab8ebd736aa16f43668e75f255b9be383b299363df895d807829c33f9 not found: ID does not exist"
Sep 10 23:50:49.229217 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.229158    2574 scope.go:117] "RemoveContainer" containerID="37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"
Sep 10 23:50:49.237446 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.237427    2574 scope.go:117] "RemoveContainer" containerID="37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"
Sep 10 23:50:49.237797 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:50:49.237734    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0\": container with ID starting with 37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0 not found: ID does not exist" containerID="37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"
Sep 10 23:50:49.237916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.237789    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0"} err="failed to get container status \"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0\": rpc error: code = NotFound desc = could not find container \"37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0\": container with ID starting with 37e7f3e013f0df5edc2f300244397a69a027674ab0fd6b412517676e56bf90d0 not found: ID does not exist"
Sep 10 23:50:49.240199 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.240178    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:49.243497 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.243477    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6a8995da-85b98df68b-lxb92"]
Sep 10 23:50:49.254522 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.254503    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:49.258423 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.258405    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6a8995da-7b769c9c86-ww6vr"]
Sep 10 23:50:49.950042 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.949970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:49.950472 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.950052    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:49.962203 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:49.962176    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:50.238204 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:50.238128    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:50:50.616105 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:50.616067    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123e15f8-23fd-4e6a-a004-6bd3e9616517" path="/var/lib/kubelet/pods/123e15f8-23fd-4e6a-a004-6bd3e9616517/volumes"
Sep 10 23:50:50.616920 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:50.616894    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" path="/var/lib/kubelet/pods/25b1392c-9ac6-4377-b62d-22e3a14d90b8/volumes"
Sep 10 23:50:50.617588 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:50.617568    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b06af46c-8fcd-480a-965e-aaf3f36acff9" path="/var/lib/kubelet/pods/b06af46c-8fcd-480a-965e-aaf3f36acff9/volumes"
Sep 10 23:50:54.434198 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434166    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:50:54.434574 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434530    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123e15f8-23fd-4e6a-a004-6bd3e9616517" containerName="revoke-keys"
Sep 10 23:50:54.434574 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434542    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="123e15f8-23fd-4e6a-a004-6bd3e9616517" containerName="revoke-keys"
Sep 10 23:50:54.434574 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434560    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b06af46c-8fcd-480a-965e-aaf3f36acff9" containerName="payload-pre-processing"
Sep 10 23:50:54.434574 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434565    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b06af46c-8fcd-480a-965e-aaf3f36acff9" containerName="payload-pre-processing"
Sep 10 23:50:54.434735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434578    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" containerName="payload-processing"
Sep 10 23:50:54.434735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434584    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" containerName="payload-processing"
Sep 10 23:50:54.434735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434634    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b1392c-9ac6-4377-b62d-22e3a14d90b8" containerName="payload-processing"
Sep 10 23:50:54.434735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434643    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b06af46c-8fcd-480a-965e-aaf3f36acff9" containerName="payload-pre-processing"
Sep 10 23:50:54.434735 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.434649    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="123e15f8-23fd-4e6a-a004-6bd3e9616517" containerName="revoke-keys"
Sep 10 23:50:54.438876 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.438859    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.441965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.441945    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5706c553\""
Sep 10 23:50:54.442178 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.442160    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5706c553-dockercfg-l72gp\""
Sep 10 23:50:54.455687 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.455666    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:50:54.464325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.464303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zvl\" (UniqueName: \"kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.464427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.464364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.484078 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.484051    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:50:54.487323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.487305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.500139 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.500117    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:50:54.565608 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.565550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.565801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.565631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.565801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.565679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqrm\" (UniqueName: \"kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.565801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.565717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zvl\" (UniqueName: \"kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.566341 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.566321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.573628 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.573601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zvl\" (UniqueName: \"kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl\") pod \"payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.666531 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.666497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.666664 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.666563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqrm\" (UniqueName: \"kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.667182 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.667163    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.674374 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.674357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqrm\" (UniqueName: \"kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm\") pod \"payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") " pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:54.748458 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.748380    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:54.797384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:54.797335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:55.017406 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.017380    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:50:55.019631 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:55.019607    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8928dc7a_8ab4_427f_b393_27dcbcaf2e23.slice/crio-a7f23dc745230c6e5904279bb868b095035d431b26322257648d50b666ab3732 WatchSource:0}: Error finding container a7f23dc745230c6e5904279bb868b095035d431b26322257648d50b666ab3732: Status 404 returned error can't find the container with id a7f23dc745230c6e5904279bb868b095035d431b26322257648d50b666ab3732
Sep 10 23:50:55.083953 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.083915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:50:55.086490 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:50:55.086460    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59074c6_61a1_413d_a69a_5d6e802a3d42.slice/crio-82c49d889edd946c94b9ae97a372c93030241fb3f31745a6b86cda8117ef7a7c WatchSource:0}: Error finding container 82c49d889edd946c94b9ae97a372c93030241fb3f31745a6b86cda8117ef7a7c: Status 404 returned error can't find the container with id 82c49d889edd946c94b9ae97a372c93030241fb3f31745a6b86cda8117ef7a7c
Sep 10 23:50:55.254575 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.254515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" event={"ID":"f59074c6-61a1-413d-a69a-5d6e802a3d42","Type":"ContainerStarted","Data":"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"}
Sep 10 23:50:55.254575 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.254577    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" event={"ID":"f59074c6-61a1-413d-a69a-5d6e802a3d42","Type":"ContainerStarted","Data":"82c49d889edd946c94b9ae97a372c93030241fb3f31745a6b86cda8117ef7a7c"}
Sep 10 23:50:55.254828 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.254687    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:50:55.256474 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.256437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" event={"ID":"8928dc7a-8ab4-427f-b393-27dcbcaf2e23","Type":"ContainerStarted","Data":"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"}
Sep 10 23:50:55.256593 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.256487    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" event={"ID":"8928dc7a-8ab4-427f-b393-27dcbcaf2e23","Type":"ContainerStarted","Data":"a7f23dc745230c6e5904279bb868b095035d431b26322257648d50b666ab3732"}
Sep 10 23:50:55.256593 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.256513    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:50:55.271341 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.271009    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" podStartSLOduration=1.270994475 podStartE2EDuration="1.270994475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:55.27057334 +0000 UTC m=+859.100400560" watchObservedRunningTime="2026-09-10 23:50:55.270994475 +0000 UTC m=+859.100821695"
Sep 10 23:50:55.286605 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:50:55.286561    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" podStartSLOduration=1.286550061 podStartE2EDuration="1.286550061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:55.28515761 +0000 UTC m=+859.114984822" watchObservedRunningTime="2026-09-10 23:50:55.286550061 +0000 UTC m=+859.116377280"
Sep 10 23:51:06.263944 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:06.263916    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:51:06.264398 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:06.263992    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:51:12.352964 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.352931    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"]
Sep 10 23:51:12.356358 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.356342    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:12.358617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.358595    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:51:12.358734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.358596    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:51:12.366360 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.366334    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"]
Sep 10 23:51:12.426203 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.426174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-74f1c1-7t444\" (UID: \"301fd1d6-4305-4367-b730-63084fb5c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:12.527532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.527499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-74f1c1-7t444\" (UID: \"301fd1d6-4305-4367-b730-63084fb5c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:12.528125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.528104    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-74f1c1-7t444\" (UID: \"301fd1d6-4305-4367-b730-63084fb5c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:12.668077 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.668008    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:12.936461 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:12.935967    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"]
Sep 10 23:51:12.938753 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:51:12.938706    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301fd1d6_4305_4367_b730_63084fb5c220.slice/crio-ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9 WatchSource:0}: Error finding container ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9: Status 404 returned error can't find the container with id ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9
Sep 10 23:51:13.348593 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:13.348561    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444" event={"ID":"301fd1d6-4305-4367-b730-63084fb5c220","Type":"ContainerStarted","Data":"e05ae244daed0f55c8ac2510f2b7ae0297331953c7f51c8128c0066c9e76869e"}
Sep 10 23:51:13.348739 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:13.348606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444" event={"ID":"301fd1d6-4305-4367-b730-63084fb5c220","Type":"ContainerStarted","Data":"ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9"}
Sep 10 23:51:13.364640 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:13.364595    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444" podStartSLOduration=1.364581257 podStartE2EDuration="1.364581257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:13.363492842 +0000 UTC m=+877.193320063" watchObservedRunningTime="2026-09-10 23:51:13.364581257 +0000 UTC m=+877.194408509"
Sep 10 23:51:14.355171 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:14.355075    2574 generic.go:356] "Generic (PLEG): container finished" podID="301fd1d6-4305-4367-b730-63084fb5c220" containerID="e05ae244daed0f55c8ac2510f2b7ae0297331953c7f51c8128c0066c9e76869e" exitCode=0
Sep 10 23:51:14.355319 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:14.355170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444" event={"ID":"301fd1d6-4305-4367-b730-63084fb5c220","Type":"ContainerDied","Data":"e05ae244daed0f55c8ac2510f2b7ae0297331953c7f51c8128c0066c9e76869e"}
Sep 10 23:51:15.486930 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:15.486907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:15.556549 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:15.556520    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca\") pod \"301fd1d6-4305-4367-b730-63084fb5c220\" (UID: \"301fd1d6-4305-4367-b730-63084fb5c220\") "
Sep 10 23:51:15.556900 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:15.556878    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "301fd1d6-4305-4367-b730-63084fb5c220" (UID: "301fd1d6-4305-4367-b730-63084fb5c220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:15.657602 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:15.657537    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/301fd1d6-4305-4367-b730-63084fb5c220-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:16.366552 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:16.366519    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"
Sep 10 23:51:16.366707 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:16.366550    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444" event={"ID":"301fd1d6-4305-4367-b730-63084fb5c220","Type":"ContainerDied","Data":"ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9"}
Sep 10 23:51:16.366707 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:16.366597    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf97e1a0ec6338b673c2e19e95cb8f45b11fff5d61bbc0af1182259f95bbef9"
Sep 10 23:51:18.040498 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.040464    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"]
Sep 10 23:51:18.041112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.041095    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301fd1d6-4305-4367-b730-63084fb5c220" containerName="revoke-keys"
Sep 10 23:51:18.041153 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.041116    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="301fd1d6-4305-4367-b730-63084fb5c220" containerName="revoke-keys"
Sep 10 23:51:18.041229 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.041218    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="301fd1d6-4305-4367-b730-63084fb5c220" containerName="revoke-keys"
Sep 10 23:51:18.044619 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.044601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:18.046932 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.046908    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:51:18.047063 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.046947    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:51:18.051831 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.051676    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"]
Sep 10 23:51:18.178949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.178916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5706c553-wn8s6\" (UID: \"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:18.279752 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.279716    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5706c553-wn8s6\" (UID: \"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:18.280323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.280303    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5706c553-wn8s6\" (UID: \"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:18.355957 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.355928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:18.624126 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:18.624103    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"]
Sep 10 23:51:18.626857 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:51:18.626831    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddff2e7_7c3d_4d1c_9160_5adbb34053ce.slice/crio-5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f WatchSource:0}: Error finding container 5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f: Status 404 returned error can't find the container with id 5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f
Sep 10 23:51:19.381734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:19.381702    2574 generic.go:356] "Generic (PLEG): container finished" podID="8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" containerID="f0092949041af49ee178a0f7fe9d39ab15ec76055c66ebb08996b5a790b8842d" exitCode=0
Sep 10 23:51:19.382057 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:19.381802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6" event={"ID":"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce","Type":"ContainerDied","Data":"f0092949041af49ee178a0f7fe9d39ab15ec76055c66ebb08996b5a790b8842d"}
Sep 10 23:51:19.382057 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:19.381845    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6" event={"ID":"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce","Type":"ContainerStarted","Data":"5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f"}
Sep 10 23:51:20.517473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:20.517448    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:20.601258 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:20.601225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca\") pod \"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce\" (UID: \"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce\") "
Sep 10 23:51:20.601572 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:20.601550    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" (UID: "8ddff2e7-7c3d-4d1c-9160-5adbb34053ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:20.702739 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:20.702656    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:21.392125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:21.392086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6" event={"ID":"8ddff2e7-7c3d-4d1c-9160-5adbb34053ce","Type":"ContainerDied","Data":"5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f"}
Sep 10 23:51:21.392125 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:21.392128    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"
Sep 10 23:51:21.392347 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:21.392145    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa6f91dc45f6fc47eeb5bab6d58aa0e6ce29da6c194efd88bf8e0997910323f"
Sep 10 23:51:23.346424 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:23.346392    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"]
Sep 10 23:51:23.350908 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:23.350885    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-74f1c1-7t444"]
Sep 10 23:51:24.577750 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.577715    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:51:24.578164 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.577995    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" podUID="f59074c6-61a1-413d-a69a-5d6e802a3d42" containerName="payload-processing" containerID="cri-o://edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe" gracePeriod=30
Sep 10 23:51:24.591308 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.591284    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:51:24.591468 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.591450    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" podUID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" containerName="payload-pre-processing" containerID="cri-o://c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa" gracePeriod=30
Sep 10 23:51:24.618730 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.618700    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301fd1d6-4305-4367-b730-63084fb5c220" path="/var/lib/kubelet/pods/301fd1d6-4305-4367-b730-63084fb5c220/volumes"
Sep 10 23:51:24.768201 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.768169    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"]
Sep 10 23:51:24.770931 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.770899    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5706c553-wn8s6"]
Sep 10 23:51:24.857134 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.857112    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:51:24.860011 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.859991    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:51:24.941457 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941420    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume\") pod \"f59074c6-61a1-413d-a69a-5d6e802a3d42\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") "
Sep 10 23:51:24.941647 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941484    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zvl\" (UniqueName: \"kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl\") pod \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") "
Sep 10 23:51:24.941647 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941626    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume\") pod \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\" (UID: \"8928dc7a-8ab4-427f-b393-27dcbcaf2e23\") "
Sep 10 23:51:24.941795 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941678    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsqrm\" (UniqueName: \"kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm\") pod \"f59074c6-61a1-413d-a69a-5d6e802a3d42\" (UID: \"f59074c6-61a1-413d-a69a-5d6e802a3d42\") "
Sep 10 23:51:24.941940 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941910    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f59074c6-61a1-413d-a69a-5d6e802a3d42" (UID: "f59074c6-61a1-413d-a69a-5d6e802a3d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:24.942070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.941962    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8928dc7a-8ab4-427f-b393-27dcbcaf2e23" (UID: "8928dc7a-8ab4-427f-b393-27dcbcaf2e23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:24.942070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.942059    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59074c6-61a1-413d-a69a-5d6e802a3d42-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:24.942155 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.942078    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:24.943660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.943636    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl" (OuterVolumeSpecName: "kube-api-access-z7zvl") pod "8928dc7a-8ab4-427f-b393-27dcbcaf2e23" (UID: "8928dc7a-8ab4-427f-b393-27dcbcaf2e23"). InnerVolumeSpecName "kube-api-access-z7zvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:24.943721 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:24.943642    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm" (OuterVolumeSpecName: "kube-api-access-fsqrm") pod "f59074c6-61a1-413d-a69a-5d6e802a3d42" (UID: "f59074c6-61a1-413d-a69a-5d6e802a3d42"). InnerVolumeSpecName "kube-api-access-fsqrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:25.042848 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.042820    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zvl\" (UniqueName: \"kubernetes.io/projected/8928dc7a-8ab4-427f-b393-27dcbcaf2e23-kube-api-access-z7zvl\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:25.042848 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.042843    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqrm\" (UniqueName: \"kubernetes.io/projected/f59074c6-61a1-413d-a69a-5d6e802a3d42-kube-api-access-fsqrm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:25.413502 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.413468    2574 generic.go:356] "Generic (PLEG): container finished" podID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" containerID="c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa" exitCode=0
Sep 10 23:51:25.413708 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.413517    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"
Sep 10 23:51:25.413708 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.413551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" event={"ID":"8928dc7a-8ab4-427f-b393-27dcbcaf2e23","Type":"ContainerDied","Data":"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"}
Sep 10 23:51:25.413708 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.413608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk" event={"ID":"8928dc7a-8ab4-427f-b393-27dcbcaf2e23","Type":"ContainerDied","Data":"a7f23dc745230c6e5904279bb868b095035d431b26322257648d50b666ab3732"}
Sep 10 23:51:25.413708 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.413628    2574 scope.go:117] "RemoveContainer" containerID="c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"
Sep 10 23:51:25.414992 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.414967    2574 generic.go:356] "Generic (PLEG): container finished" podID="f59074c6-61a1-413d-a69a-5d6e802a3d42" containerID="edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe" exitCode=0
Sep 10 23:51:25.415079 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.415004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" event={"ID":"f59074c6-61a1-413d-a69a-5d6e802a3d42","Type":"ContainerDied","Data":"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"}
Sep 10 23:51:25.415079 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.415055    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx" event={"ID":"f59074c6-61a1-413d-a69a-5d6e802a3d42","Type":"ContainerDied","Data":"82c49d889edd946c94b9ae97a372c93030241fb3f31745a6b86cda8117ef7a7c"}
Sep 10 23:51:25.415238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.415221    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"
Sep 10 23:51:25.424300 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.424280    2574 scope.go:117] "RemoveContainer" containerID="c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"
Sep 10 23:51:25.424582 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:51:25.424565    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa\": container with ID starting with c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa not found: ID does not exist" containerID="c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"
Sep 10 23:51:25.424656 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.424594    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa"} err="failed to get container status \"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa\": rpc error: code = NotFound desc = could not find container \"c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa\": container with ID starting with c39417cec01938671eafad5bb5f105b3ef586610d0bfb0068bda7089c8ca71aa not found: ID does not exist"
Sep 10 23:51:25.424656 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.424617    2574 scope.go:117] "RemoveContainer" containerID="edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"
Sep 10 23:51:25.432360 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.432344    2574 scope.go:117] "RemoveContainer" containerID="edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"
Sep 10 23:51:25.432615 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:51:25.432599    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe\": container with ID starting with edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe not found: ID does not exist" containerID="edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"
Sep 10 23:51:25.432675 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.432622    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe"} err="failed to get container status \"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe\": rpc error: code = NotFound desc = could not find container \"edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe\": container with ID starting with edd758f7aebc22be7f9f629c148eeb7bca17d3d6239c4d3dc2c4f275ac783dfe not found: ID does not exist"
Sep 10 23:51:25.436920 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.436898    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:51:25.440157 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.440135    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5706c553-799c756c9d-959bk"]
Sep 10 23:51:25.449823 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.449800    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:51:25.453603 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:25.453583    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5706c553-85cb5d8b9f-4nfkx"]
Sep 10 23:51:26.616190 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:26.616157    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" path="/var/lib/kubelet/pods/8928dc7a-8ab4-427f-b393-27dcbcaf2e23/volumes"
Sep 10 23:51:26.616556 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:26.616485    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" path="/var/lib/kubelet/pods/8ddff2e7-7c3d-4d1c-9160-5adbb34053ce/volumes"
Sep 10 23:51:26.616827 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:26.616807    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59074c6-61a1-413d-a69a-5d6e802a3d42" path="/var/lib/kubelet/pods/f59074c6-61a1-413d-a69a-5d6e802a3d42/volumes"
Sep 10 23:51:28.685274 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:28.685238    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:51:43.704579 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:43.704534    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" podUID="71b6652b-94eb-488a-ab83-d083bccf5301" containerName="main" containerID="cri-o://640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48" gracePeriod=45
Sep 10 23:51:43.964144 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:43.964092    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:51:44.009536 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.009703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009569    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.009703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009614    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.009703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009654    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn5ln\" (UniqueName: \"kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.009703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009689    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.009962 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009758    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir\") pod \"71b6652b-94eb-488a-ab83-d083bccf5301\" (UID: \"71b6652b-94eb-488a-ab83-d083bccf5301\") "
Sep 10 23:51:44.010017 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.009992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home" (OuterVolumeSpecName: "home") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:51:44.010223 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.010181    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:51:44.010223 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.010209    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache" (OuterVolumeSpecName: "model-cache") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:51:44.012154 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.012121    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm" (OuterVolumeSpecName: "dshm") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:51:44.012238 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.012183    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln" (OuterVolumeSpecName: "kube-api-access-hn5ln") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "kube-api-access-hn5ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:51:44.012292 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.012277    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71b6652b-94eb-488a-ab83-d083bccf5301" (UID: "71b6652b-94eb-488a-ab83-d083bccf5301"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:51:44.111269 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111241    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.111269 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111265    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b6652b-94eb-488a-ab83-d083bccf5301-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.111419 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111275    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.111419 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111283    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.111419 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111292    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5ln\" (UniqueName: \"kubernetes.io/projected/71b6652b-94eb-488a-ab83-d083bccf5301-kube-api-access-hn5ln\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.111419 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.111300    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b6652b-94eb-488a-ab83-d083bccf5301-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:44.505447 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.505416    2574 generic.go:356] "Generic (PLEG): container finished" podID="71b6652b-94eb-488a-ab83-d083bccf5301" containerID="640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48" exitCode=0
Sep 10 23:51:44.505617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.505459    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"
Sep 10 23:51:44.505617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.505504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" event={"ID":"71b6652b-94eb-488a-ab83-d083bccf5301","Type":"ContainerDied","Data":"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"}
Sep 10 23:51:44.505617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.505549    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8" event={"ID":"71b6652b-94eb-488a-ab83-d083bccf5301","Type":"ContainerDied","Data":"49d39c977c873345ecdd725dc5fcbb5139a5e7c217052dc628b39647698965f5"}
Sep 10 23:51:44.505617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.505571    2574 scope.go:117] "RemoveContainer" containerID="640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"
Sep 10 23:51:44.514435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.514420    2574 scope.go:117] "RemoveContainer" containerID="640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"
Sep 10 23:51:44.514695 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:51:44.514678    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48\": container with ID starting with 640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48 not found: ID does not exist" containerID="640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"
Sep 10 23:51:44.514790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.514700    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48"} err="failed to get container status \"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48\": rpc error: code = NotFound desc = could not find container \"640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48\": container with ID starting with 640be1f5650f54029a02228a9f0e6ab58bc12557ad625d9936acb650641b1e48 not found: ID does not exist"
Sep 10 23:51:44.527449 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.527423    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:51:44.533675 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.533656    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-74f1c1/autores-model-74f1c1-kserve-5bd6f4767-cqsg8"]
Sep 10 23:51:44.614111 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:44.614085    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b6652b-94eb-488a-ab83-d083bccf5301" path="/var/lib/kubelet/pods/71b6652b-94eb-488a-ab83-d083bccf5301/volumes"
Sep 10 23:51:54.405027 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.404997    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"]
Sep 10 23:51:54.405535 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405515    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59074c6-61a1-413d-a69a-5d6e802a3d42" containerName="payload-processing"
Sep 10 23:51:54.405581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405540    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59074c6-61a1-413d-a69a-5d6e802a3d42" containerName="payload-processing"
Sep 10 23:51:54.405581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405560    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b6652b-94eb-488a-ab83-d083bccf5301" containerName="main"
Sep 10 23:51:54.405581 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405568    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b6652b-94eb-488a-ab83-d083bccf5301" containerName="main"
Sep 10 23:51:54.405690 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405586    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" containerName="payload-pre-processing"
Sep 10 23:51:54.405690 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405596    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" containerName="payload-pre-processing"
Sep 10 23:51:54.405690 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405607    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" containerName="revoke-keys"
Sep 10 23:51:54.405690 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405616    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" containerName="revoke-keys"
Sep 10 23:51:54.405898 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405709    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8928dc7a-8ab4-427f-b393-27dcbcaf2e23" containerName="payload-pre-processing"
Sep 10 23:51:54.405898 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405723    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddff2e7-7c3d-4d1c-9160-5adbb34053ce" containerName="revoke-keys"
Sep 10 23:51:54.405898 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405736    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b6652b-94eb-488a-ab83-d083bccf5301" containerName="main"
Sep 10 23:51:54.405898 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.405748    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59074c6-61a1-413d-a69a-5d6e802a3d42" containerName="payload-processing"
Sep 10 23:51:54.409932 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.409914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:54.412272 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.412251    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:51:54.412373 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.412307    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:51:54.416347 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.416325    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"]
Sep 10 23:51:54.502594 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.502563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r\" (UID: \"c8641467-8dba-47bf-ae86-ca88a6d94812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:54.603559 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.603525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r\" (UID: \"c8641467-8dba-47bf-ae86-ca88a6d94812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:54.604138 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.604121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r\" (UID: \"c8641467-8dba-47bf-ae86-ca88a6d94812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:54.721435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.721352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:54.995240 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:54.995215    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"]
Sep 10 23:51:54.997704 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:51:54.997670    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8641467_8dba_47bf_ae86_ca88a6d94812.slice/crio-b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b WatchSource:0}: Error finding container b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b: Status 404 returned error can't find the container with id b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b
Sep 10 23:51:55.566128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:55.566092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r" event={"ID":"c8641467-8dba-47bf-ae86-ca88a6d94812","Type":"ContainerStarted","Data":"39fec19cda335936c2851c3db68a8a20ebe7c23fd0cdaf1f7711e50d0af3ed4c"}
Sep 10 23:51:55.566617 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:55.566145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r" event={"ID":"c8641467-8dba-47bf-ae86-ca88a6d94812","Type":"ContainerStarted","Data":"b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b"}
Sep 10 23:51:55.582104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:55.582050    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r" podStartSLOduration=1.582032195 podStartE2EDuration="1.582032195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:55.581353414 +0000 UTC m=+919.411180633" watchObservedRunningTime="2026-09-10 23:51:55.582032195 +0000 UTC m=+919.411859421"
Sep 10 23:51:56.573594 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:56.573559    2574 generic.go:356] "Generic (PLEG): container finished" podID="c8641467-8dba-47bf-ae86-ca88a6d94812" containerID="39fec19cda335936c2851c3db68a8a20ebe7c23fd0cdaf1f7711e50d0af3ed4c" exitCode=0
Sep 10 23:51:56.573965 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:56.573633    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r" event={"ID":"c8641467-8dba-47bf-ae86-ca88a6d94812","Type":"ContainerDied","Data":"39fec19cda335936c2851c3db68a8a20ebe7c23fd0cdaf1f7711e50d0af3ed4c"}
Sep 10 23:51:57.704050 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:57.704025    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:57.827349 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:57.827320    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca\") pod \"c8641467-8dba-47bf-ae86-ca88a6d94812\" (UID: \"c8641467-8dba-47bf-ae86-ca88a6d94812\") "
Sep 10 23:51:57.827658 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:57.827636    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8641467-8dba-47bf-ae86-ca88a6d94812" (UID: "c8641467-8dba-47bf-ae86-ca88a6d94812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:51:57.928992 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:57.928956    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8641467-8dba-47bf-ae86-ca88a6d94812-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:51:58.584848 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:58.584806    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r" event={"ID":"c8641467-8dba-47bf-ae86-ca88a6d94812","Type":"ContainerDied","Data":"b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b"}
Sep 10 23:51:58.584848 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:58.584845    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"
Sep 10 23:51:58.585064 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:51:58.584894    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0eeb6458255d1e1925e3bb68243e26e50266c69cc02ea51157256dd6ea74f5b"
Sep 10 23:52:01.462615 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:01.462573    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"]
Sep 10 23:52:01.465522 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:01.465495    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4b2fb82-2xh4r"]
Sep 10 23:52:02.618688 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:02.618648    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8641467-8dba-47bf-ae86-ca88a6d94812" path="/var/lib/kubelet/pods/c8641467-8dba-47bf-ae86-ca88a6d94812/volumes"
Sep 10 23:52:44.601852 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.601817    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:52:44.602261 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.602147    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8641467-8dba-47bf-ae86-ca88a6d94812" containerName="revoke-keys"
Sep 10 23:52:44.602261 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.602158    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8641467-8dba-47bf-ae86-ca88a6d94812" containerName="revoke-keys"
Sep 10 23:52:44.602261 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.602221    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8641467-8dba-47bf-ae86-ca88a6d94812" containerName="revoke-keys"
Sep 10 23:52:44.606552 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.606530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.609462 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.609441    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b88386\""
Sep 10 23:52:44.628267 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.628234    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:52:44.702715 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.702680    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:52:44.706079 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.706063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.708279 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.708257    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b88386-dockercfg-84cck\""
Sep 10 23:52:44.719567 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.719545    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:52:44.741333 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.741309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzwv\" (UniqueName: \"kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.741450 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.741336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.841889 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.841858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4km7n\" (UniqueName: \"kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.841889 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.841893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.842124 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.841936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzwv\" (UniqueName: \"kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.842124 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.841957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.842628 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.842608    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.850193 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.850169    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzwv\" (UniqueName: \"kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv\") pod \"payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.917163 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.917102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:44.943356 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.943331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4km7n\" (UniqueName: \"kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.943471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.943363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.944050 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.944027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:44.957719 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:44.957691    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4km7n\" (UniqueName: \"kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n\") pod \"payload-processing-e2e-explicit-b88386-5976c9c-vh2qj\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:45.016454 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.016421    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:45.231109 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.231079    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:52:45.232904 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:52:45.232876    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547bd370_d929_41fb_b6e4_b93f61453a3b.slice/crio-1e8341f478001e47cd852b6c8a05ea840e6117b740c4505d2190a61888230479 WatchSource:0}: Error finding container 1e8341f478001e47cd852b6c8a05ea840e6117b740c4505d2190a61888230479: Status 404 returned error can't find the container with id 1e8341f478001e47cd852b6c8a05ea840e6117b740c4505d2190a61888230479
Sep 10 23:52:45.315305 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.315279    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:52:45.316569 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:52:45.316546    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c56386_cd74_4d45_81e2_722522373ff5.slice/crio-df66a7f315bfdf6feec68a02730a83e1f60300955335d0a0a8a643650f46a388 WatchSource:0}: Error finding container df66a7f315bfdf6feec68a02730a83e1f60300955335d0a0a8a643650f46a388: Status 404 returned error can't find the container with id df66a7f315bfdf6feec68a02730a83e1f60300955335d0a0a8a643650f46a388
Sep 10 23:52:45.813332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.813293    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" event={"ID":"547bd370-d929-41fb-b6e4-b93f61453a3b","Type":"ContainerStarted","Data":"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"}
Sep 10 23:52:45.813812 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.813389    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" event={"ID":"547bd370-d929-41fb-b6e4-b93f61453a3b","Type":"ContainerStarted","Data":"1e8341f478001e47cd852b6c8a05ea840e6117b740c4505d2190a61888230479"}
Sep 10 23:52:45.813812 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.813419    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:52:45.814937 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.814899    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" event={"ID":"a2c56386-cd74-4d45-81e2-722522373ff5","Type":"ContainerStarted","Data":"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"}
Sep 10 23:52:45.815093 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.814951    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" event={"ID":"a2c56386-cd74-4d45-81e2-722522373ff5","Type":"ContainerStarted","Data":"df66a7f315bfdf6feec68a02730a83e1f60300955335d0a0a8a643650f46a388"}
Sep 10 23:52:45.815093 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.815050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:45.830782 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.830723    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" podStartSLOduration=1.830708074 podStartE2EDuration="1.830708074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:45.828855864 +0000 UTC m=+969.658683083" watchObservedRunningTime="2026-09-10 23:52:45.830708074 +0000 UTC m=+969.660535340"
Sep 10 23:52:45.846427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:45.846248    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" podStartSLOduration=1.846234148 podStartE2EDuration="1.846234148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:45.845782863 +0000 UTC m=+969.675610084" watchObservedRunningTime="2026-09-10 23:52:45.846234148 +0000 UTC m=+969.676061370"
Sep 10 23:52:56.821133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:56.821103    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:52:56.821532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:52:56.821178    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:53:49.040084 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.040051    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"]
Sep 10 23:53:49.043445 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.043426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:49.045706 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.045690    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:53:49.045827 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.045806    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:53:49.051243 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.051209    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"]
Sep 10 23:53:49.219516 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.219476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b88386-d8js8\" (UID: \"3cabbeaf-5efe-4437-a7d1-3e5039fba86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:49.320470 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.320390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b88386-d8js8\" (UID: \"3cabbeaf-5efe-4437-a7d1-3e5039fba86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:49.321030 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.321011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b88386-d8js8\" (UID: \"3cabbeaf-5efe-4437-a7d1-3e5039fba86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:49.354779 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.354739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:49.624047 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:49.624022    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"]
Sep 10 23:53:49.626038 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:53:49.626004    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cabbeaf_5efe_4437_a7d1_3e5039fba86e.slice/crio-3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388 WatchSource:0}: Error finding container 3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388: Status 404 returned error can't find the container with id 3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388
Sep 10 23:53:50.131151 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:50.131115    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8" event={"ID":"3cabbeaf-5efe-4437-a7d1-3e5039fba86e","Type":"ContainerStarted","Data":"249304f866f525184565bfb41b161ebd85c3134c5487b5acba11b17b4eac537c"}
Sep 10 23:53:50.131500 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:50.131167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8" event={"ID":"3cabbeaf-5efe-4437-a7d1-3e5039fba86e","Type":"ContainerStarted","Data":"3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388"}
Sep 10 23:53:50.146218 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:50.146165    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8" podStartSLOduration=1.14614621 podStartE2EDuration="1.14614621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:50.145277351 +0000 UTC m=+1033.975104570" watchObservedRunningTime="2026-09-10 23:53:50.14614621 +0000 UTC m=+1033.975973430"
Sep 10 23:53:51.137729 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:51.137698    2574 generic.go:356] "Generic (PLEG): container finished" podID="3cabbeaf-5efe-4437-a7d1-3e5039fba86e" containerID="249304f866f525184565bfb41b161ebd85c3134c5487b5acba11b17b4eac537c" exitCode=0
Sep 10 23:53:51.138118 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:51.137802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8" event={"ID":"3cabbeaf-5efe-4437-a7d1-3e5039fba86e","Type":"ContainerDied","Data":"249304f866f525184565bfb41b161ebd85c3134c5487b5acba11b17b4eac537c"}
Sep 10 23:53:52.272937 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:52.272914    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:52.448102 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:52.448021    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca\") pod \"3cabbeaf-5efe-4437-a7d1-3e5039fba86e\" (UID: \"3cabbeaf-5efe-4437-a7d1-3e5039fba86e\") "
Sep 10 23:53:52.448392 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:52.448364    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cabbeaf-5efe-4437-a7d1-3e5039fba86e" (UID: "3cabbeaf-5efe-4437-a7d1-3e5039fba86e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:53:52.549048 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:52.549015    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cabbeaf-5efe-4437-a7d1-3e5039fba86e-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:53:53.149582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:53.149551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8" event={"ID":"3cabbeaf-5efe-4437-a7d1-3e5039fba86e","Type":"ContainerDied","Data":"3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388"}
Sep 10 23:53:53.149582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:53.149577    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"
Sep 10 23:53:53.149582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:53.149597    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3528435920d0ec63b594ab042fee6fe50ceab0d3a5c894c34d2bcddd004b4388"
Sep 10 23:53:59.553308 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.553272    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:53:59.553716 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.553513    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" podUID="a2c56386-cd74-4d45-81e2-722522373ff5" containerName="payload-processing" containerID="cri-o://21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8" gracePeriod=30
Sep 10 23:53:59.564476 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.564417    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:53:59.564709 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.564657    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" podUID="547bd370-d929-41fb-b6e4-b93f61453a3b" containerName="payload-pre-processing" containerID="cri-o://3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30" gracePeriod=30
Sep 10 23:53:59.720708 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.720668    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"]
Sep 10 23:53:59.722588 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.722557    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b88386-d8js8"]
Sep 10 23:53:59.821435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.821404    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:53:59.852470 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.852445    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:53:59.913554 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913524    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume\") pod \"a2c56386-cd74-4d45-81e2-722522373ff5\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") "
Sep 10 23:53:59.913734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume\") pod \"547bd370-d929-41fb-b6e4-b93f61453a3b\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") "
Sep 10 23:53:59.913734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913582    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhzwv\" (UniqueName: \"kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv\") pod \"547bd370-d929-41fb-b6e4-b93f61453a3b\" (UID: \"547bd370-d929-41fb-b6e4-b93f61453a3b\") "
Sep 10 23:53:59.913734 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913616    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4km7n\" (UniqueName: \"kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n\") pod \"a2c56386-cd74-4d45-81e2-722522373ff5\" (UID: \"a2c56386-cd74-4d45-81e2-722522373ff5\") "
Sep 10 23:53:59.914016 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "547bd370-d929-41fb-b6e4-b93f61453a3b" (UID: "547bd370-d929-41fb-b6e4-b93f61453a3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:53:59.914066 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.913994    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2c56386-cd74-4d45-81e2-722522373ff5" (UID: "a2c56386-cd74-4d45-81e2-722522373ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:53:59.915713 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.915689    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n" (OuterVolumeSpecName: "kube-api-access-4km7n") pod "a2c56386-cd74-4d45-81e2-722522373ff5" (UID: "a2c56386-cd74-4d45-81e2-722522373ff5"). InnerVolumeSpecName "kube-api-access-4km7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:53:59.915713 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:53:59.915707    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv" (OuterVolumeSpecName: "kube-api-access-jhzwv") pod "547bd370-d929-41fb-b6e4-b93f61453a3b" (UID: "547bd370-d929-41fb-b6e4-b93f61453a3b"). InnerVolumeSpecName "kube-api-access-jhzwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:00.014839 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.014803    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c56386-cd74-4d45-81e2-722522373ff5-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.014839 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.014835    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547bd370-d929-41fb-b6e4-b93f61453a3b-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.014839 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.014847    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzwv\" (UniqueName: \"kubernetes.io/projected/547bd370-d929-41fb-b6e4-b93f61453a3b-kube-api-access-jhzwv\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.015060 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.014860    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4km7n\" (UniqueName: \"kubernetes.io/projected/a2c56386-cd74-4d45-81e2-722522373ff5-kube-api-access-4km7n\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:00.186207 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.186169    2574 generic.go:356] "Generic (PLEG): container finished" podID="a2c56386-cd74-4d45-81e2-722522373ff5" containerID="21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8" exitCode=0
Sep 10 23:54:00.186411 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.186217    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"
Sep 10 23:54:00.186411 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.186252    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" event={"ID":"a2c56386-cd74-4d45-81e2-722522373ff5","Type":"ContainerDied","Data":"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"}
Sep 10 23:54:00.186411 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.186301    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj" event={"ID":"a2c56386-cd74-4d45-81e2-722522373ff5","Type":"ContainerDied","Data":"df66a7f315bfdf6feec68a02730a83e1f60300955335d0a0a8a643650f46a388"}
Sep 10 23:54:00.186411 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.186327    2574 scope.go:117] "RemoveContainer" containerID="21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"
Sep 10 23:54:00.187729 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.187705    2574 generic.go:356] "Generic (PLEG): container finished" podID="547bd370-d929-41fb-b6e4-b93f61453a3b" containerID="3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30" exitCode=0
Sep 10 23:54:00.187872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.187735    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" event={"ID":"547bd370-d929-41fb-b6e4-b93f61453a3b","Type":"ContainerDied","Data":"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"}
Sep 10 23:54:00.187872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.187757    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"
Sep 10 23:54:00.187872 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.187818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh" event={"ID":"547bd370-d929-41fb-b6e4-b93f61453a3b","Type":"ContainerDied","Data":"1e8341f478001e47cd852b6c8a05ea840e6117b740c4505d2190a61888230479"}
Sep 10 23:54:00.196700 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.196681    2574 scope.go:117] "RemoveContainer" containerID="21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"
Sep 10 23:54:00.197042 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:00.197024    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8\": container with ID starting with 21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8 not found: ID does not exist" containerID="21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"
Sep 10 23:54:00.197128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.197049    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8"} err="failed to get container status \"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8\": rpc error: code = NotFound desc = could not find container \"21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8\": container with ID starting with 21bed08c563acfffba07e19d5eb110f83ce076503b54b1a3bfbec3bddb09eae8 not found: ID does not exist"
Sep 10 23:54:00.197128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.197068    2574 scope.go:117] "RemoveContainer" containerID="3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"
Sep 10 23:54:00.205084 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.205065    2574 scope.go:117] "RemoveContainer" containerID="3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"
Sep 10 23:54:00.205363 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:00.205345    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30\": container with ID starting with 3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30 not found: ID does not exist" containerID="3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"
Sep 10 23:54:00.205436 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.205372    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30"} err="failed to get container status \"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30\": rpc error: code = NotFound desc = could not find container \"3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30\": container with ID starting with 3038509be3db4c5d4f5057da0a9ca2f8845e0a6574d4324e79980e90a7e2ad30 not found: ID does not exist"
Sep 10 23:54:00.208233 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.208210    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:54:00.212084 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.212063    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b88386-5976c9c-vh2qj"]
Sep 10 23:54:00.221335 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.221306    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:54:00.225863 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.225842    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b88386-c7cdc6858-f9kzh"]
Sep 10 23:54:00.615250 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.615219    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cabbeaf-5efe-4437-a7d1-3e5039fba86e" path="/var/lib/kubelet/pods/3cabbeaf-5efe-4437-a7d1-3e5039fba86e/volumes"
Sep 10 23:54:00.615598 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.615585    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547bd370-d929-41fb-b6e4-b93f61453a3b" path="/var/lib/kubelet/pods/547bd370-d929-41fb-b6e4-b93f61453a3b/volumes"
Sep 10 23:54:00.615930 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:00.615918    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c56386-cd74-4d45-81e2-722522373ff5" path="/var/lib/kubelet/pods/a2c56386-cd74-4d45-81e2-722522373ff5/volumes"
Sep 10 23:54:07.440898 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.440866    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:07.441379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441361    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c56386-cd74-4d45-81e2-722522373ff5" containerName="payload-processing"
Sep 10 23:54:07.441459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441382    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c56386-cd74-4d45-81e2-722522373ff5" containerName="payload-processing"
Sep 10 23:54:07.441459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441402    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547bd370-d929-41fb-b6e4-b93f61453a3b" containerName="payload-pre-processing"
Sep 10 23:54:07.441459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441411    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="547bd370-d929-41fb-b6e4-b93f61453a3b" containerName="payload-pre-processing"
Sep 10 23:54:07.441459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441425    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cabbeaf-5efe-4437-a7d1-3e5039fba86e" containerName="revoke-keys"
Sep 10 23:54:07.441459 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441434    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cabbeaf-5efe-4437-a7d1-3e5039fba86e" containerName="revoke-keys"
Sep 10 23:54:07.441691 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441532    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cabbeaf-5efe-4437-a7d1-3e5039fba86e" containerName="revoke-keys"
Sep 10 23:54:07.441691 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441546    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c56386-cd74-4d45-81e2-722522373ff5" containerName="payload-processing"
Sep 10 23:54:07.441691 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.441557    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="547bd370-d929-41fb-b6e4-b93f61453a3b" containerName="payload-pre-processing"
Sep 10 23:54:07.444412 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.444392    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.447301 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.447279    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d43496-openshift-default-dockercfg-8hzqb\""
Sep 10 23:54:07.460377 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.460352    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:07.480508 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480800 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480696    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls52\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480800 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480751    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480905 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.480905 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.481004 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.480909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.581880 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls52\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.581880 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.581993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582115 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582415 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582415 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582607 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582664 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.582962 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.582913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.584404 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.584386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.584644 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.584625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.593888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.593865    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.594362 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.594346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls52\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52\") pod \"e2e-noten-d43496-openshift-default-664b847b88-rdj86\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") " pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:07.754427 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:07.754345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:08.049223 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.049198    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:08.051318 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:54:08.051285    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdf9d81_360c_46a4_8f36_d7979c918535.slice/crio-e32ae66bff656795cd9ecaffd2824691f9fd53868224a3f1f786106d6bd46c35 WatchSource:0}: Error finding container e32ae66bff656795cd9ecaffd2824691f9fd53868224a3f1f786106d6bd46c35: Status 404 returned error can't find the container with id e32ae66bff656795cd9ecaffd2824691f9fd53868224a3f1f786106d6bd46c35
Sep 10 23:54:08.053648 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.053613    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:54:08.053854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.053699    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:54:08.053854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.053736    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:54:08.233733 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.233692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" event={"ID":"dbdf9d81-360c-46a4-8f36-d7979c918535","Type":"ContainerStarted","Data":"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"}
Sep 10 23:54:08.233904 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.233756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" event={"ID":"dbdf9d81-360c-46a4-8f36-d7979c918535","Type":"ContainerStarted","Data":"e32ae66bff656795cd9ecaffd2824691f9fd53868224a3f1f786106d6bd46c35"}
Sep 10 23:54:08.257705 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.257650    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" podStartSLOduration=1.25762806 podStartE2EDuration="1.25762806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:08.254948825 +0000 UTC m=+1052.084776044" watchObservedRunningTime="2026-09-10 23:54:08.25762806 +0000 UTC m=+1052.087455280"
Sep 10 23:54:08.755066 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:08.755035    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:09.760004 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:09.759973    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:10.243610 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:10.243578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:10.244743 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:10.244724    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:13.350650 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.350618    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"]
Sep 10 23:54:13.354121 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.354104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:13.356565 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.356544    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:54:13.356668 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.356544    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:54:13.368073 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.368052    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"]
Sep 10 23:54:13.435711 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.435681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-24749012-b4zxx\" (UID: \"dcb88095-4146-4c50-9360-6c3c7f793021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:13.536302 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.536273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-24749012-b4zxx\" (UID: \"dcb88095-4146-4c50-9360-6c3c7f793021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:13.536903 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.536885    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-24749012-b4zxx\" (UID: \"dcb88095-4146-4c50-9360-6c3c7f793021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:13.664532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.664460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:13.935488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:13.935461    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"]
Sep 10 23:54:13.937576 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:54:13.937552    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb88095_4146_4c50_9360_6c3c7f793021.slice/crio-f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9 WatchSource:0}: Error finding container f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9: Status 404 returned error can't find the container with id f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9
Sep 10 23:54:14.263844 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:14.263805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx" event={"ID":"dcb88095-4146-4c50-9360-6c3c7f793021","Type":"ContainerStarted","Data":"4f3ebea878c05eaae612383de88f9c4b897165e05152e43ef1db97fcb61c54a9"}
Sep 10 23:54:14.263974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:14.263864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx" event={"ID":"dcb88095-4146-4c50-9360-6c3c7f793021","Type":"ContainerStarted","Data":"f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9"}
Sep 10 23:54:14.279234 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:14.279188    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx" podStartSLOduration=1.279172226 podStartE2EDuration="1.279172226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:14.278296935 +0000 UTC m=+1058.108124168" watchObservedRunningTime="2026-09-10 23:54:14.279172226 +0000 UTC m=+1058.108999445"
Sep 10 23:54:15.269571 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:15.269537    2574 generic.go:356] "Generic (PLEG): container finished" podID="dcb88095-4146-4c50-9360-6c3c7f793021" containerID="4f3ebea878c05eaae612383de88f9c4b897165e05152e43ef1db97fcb61c54a9" exitCode=0
Sep 10 23:54:15.269950 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:15.269614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx" event={"ID":"dcb88095-4146-4c50-9360-6c3c7f793021","Type":"ContainerDied","Data":"4f3ebea878c05eaae612383de88f9c4b897165e05152e43ef1db97fcb61c54a9"}
Sep 10 23:54:16.402681 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:16.402656    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:16.461000 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:16.460975    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca\") pod \"dcb88095-4146-4c50-9360-6c3c7f793021\" (UID: \"dcb88095-4146-4c50-9360-6c3c7f793021\") "
Sep 10 23:54:16.461332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:16.461304    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcb88095-4146-4c50-9360-6c3c7f793021" (UID: "dcb88095-4146-4c50-9360-6c3c7f793021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:16.561783 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:16.561693    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb88095-4146-4c50-9360-6c3c7f793021-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:17.280063 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:17.280032    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"
Sep 10 23:54:17.280233 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:17.280071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx" event={"ID":"dcb88095-4146-4c50-9360-6c3c7f793021","Type":"ContainerDied","Data":"f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9"}
Sep 10 23:54:17.280233 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:17.280125    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f527d822739334a959074f5f49dcdf809c61baa34589002af24f15174e3361a9"
Sep 10 23:54:23.929316 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.929282    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:23.929951 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.929638    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb88095-4146-4c50-9360-6c3c7f793021" containerName="revoke-keys"
Sep 10 23:54:23.929951 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.929649    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb88095-4146-4c50-9360-6c3c7f793021" containerName="revoke-keys"
Sep 10 23:54:23.929951 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.929720    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb88095-4146-4c50-9360-6c3c7f793021" containerName="revoke-keys"
Sep 10 23:54:23.932990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.932969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:23.935805 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.935751    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d43496\"/\"noten-model-d43496-kserve-self-signed-certs\""
Sep 10 23:54:23.936529 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.936503    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d43496\"/\"openshift-service-ca.crt\""
Sep 10 23:54:23.936676 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.936508    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d43496\"/\"kube-root-ca.crt\""
Sep 10 23:54:23.936676 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.936563    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d43496\"/\"default-dockercfg-skctm\""
Sep 10 23:54:23.944366 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:23.944338    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:24.023348 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.023618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.023786 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.023905 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2tl\" (UniqueName: \"kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.023905 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.024031 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.023934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.124748 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.124701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.124953 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.124783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.124953 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.124904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.124953 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.124937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2tl\" (UniqueName: \"kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.125128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.124965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.125128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.125047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.125128 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.125117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.125282 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.125252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.125426 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.125377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.127495 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.127468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.127628 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.127468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.133741 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.133712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2tl\" (UniqueName: \"kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl\") pod \"noten-model-d43496-kserve-5bd75b69bc-nqq5z\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") " pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.172788 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.172728    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"]
Sep 10 23:54:24.175657 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.175632    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-24749012-b4zxx"]
Sep 10 23:54:24.245760 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.245677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:24.528961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.528874    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:24.532424 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:54:24.532386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4dd248_f6f2_4d67_92f0_edfb5fa0f3d8.slice/crio-c2dc009e6fde299a6eb76d99249313e53aee3114b7b330d042f0fb2cc1f756c1 WatchSource:0}: Error finding container c2dc009e6fde299a6eb76d99249313e53aee3114b7b330d042f0fb2cc1f756c1: Status 404 returned error can't find the container with id c2dc009e6fde299a6eb76d99249313e53aee3114b7b330d042f0fb2cc1f756c1
Sep 10 23:54:24.615575 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:24.615546    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb88095-4146-4c50-9360-6c3c7f793021" path="/var/lib/kubelet/pods/dcb88095-4146-4c50-9360-6c3c7f793021/volumes"
Sep 10 23:54:25.321054 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:25.321013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" event={"ID":"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8","Type":"ContainerStarted","Data":"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"}
Sep 10 23:54:25.321054 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:25.321059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" event={"ID":"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8","Type":"ContainerStarted","Data":"c2dc009e6fde299a6eb76d99249313e53aee3114b7b330d042f0fb2cc1f756c1"}
Sep 10 23:54:25.339982 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:25.339902    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" podStartSLOduration=2.339887342 podStartE2EDuration="2.339887342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:25.337860025 +0000 UTC m=+1069.167687244" watchObservedRunningTime="2026-09-10 23:54:25.339887342 +0000 UTC m=+1069.169714561"
Sep 10 23:54:28.344103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:28.344066    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:28.445460 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:28.445415    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:28.445709 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:28.445683    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" podUID="dbdf9d81-360c-46a4-8f36-d7979c918535" containerName="istio-proxy" containerID="cri-o://1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176" gracePeriod=30
Sep 10 23:54:29.250852 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.250760    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"]
Sep 10 23:54:29.254430 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.254411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:29.256693 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.256670    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:54:29.256851 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.256743    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:54:29.263425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.263404    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"]
Sep 10 23:54:29.377078 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.377045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5\" (UID: \"88d35889-42c6-43f9-b8d7-368624a23dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:29.478546 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.478511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5\" (UID: \"88d35889-42c6-43f9-b8d7-368624a23dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:29.479149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.479129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5\" (UID: \"88d35889-42c6-43f9-b8d7-368624a23dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:29.566132 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.566054    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:29.836678 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:29.836652    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"]
Sep 10 23:54:29.839386 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:54:29.839358    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d35889_42c6_43f9_b8d7_368624a23dfa.slice/crio-d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a WatchSource:0}: Error finding container d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a: Status 404 returned error can't find the container with id d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a
Sep 10 23:54:30.348562 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:30.348530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5" event={"ID":"88d35889-42c6-43f9-b8d7-368624a23dfa","Type":"ContainerStarted","Data":"03f0f946609e11a9d090108dfb2273b261bce0891576ed9e6efbdadbbd0b5fee"}
Sep 10 23:54:30.348733 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:30.348580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5" event={"ID":"88d35889-42c6-43f9-b8d7-368624a23dfa","Type":"ContainerStarted","Data":"d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a"}
Sep 10 23:54:30.363704 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:30.363646    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5" podStartSLOduration=1.363625607 podStartE2EDuration="1.363625607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:30.363335908 +0000 UTC m=+1074.193163128" watchObservedRunningTime="2026-09-10 23:54:30.363625607 +0000 UTC m=+1074.193452827"
Sep 10 23:54:31.354532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:31.354499    2574 generic.go:356] "Generic (PLEG): container finished" podID="88d35889-42c6-43f9-b8d7-368624a23dfa" containerID="03f0f946609e11a9d090108dfb2273b261bce0891576ed9e6efbdadbbd0b5fee" exitCode=0
Sep 10 23:54:31.354532 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:31.354537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5" event={"ID":"88d35889-42c6-43f9-b8d7-368624a23dfa","Type":"ContainerDied","Data":"03f0f946609e11a9d090108dfb2273b261bce0891576ed9e6efbdadbbd0b5fee"}
Sep 10 23:54:32.490133 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:32.490107    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:32.609072 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:32.609038    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca\") pod \"88d35889-42c6-43f9-b8d7-368624a23dfa\" (UID: \"88d35889-42c6-43f9-b8d7-368624a23dfa\") "
Sep 10 23:54:32.609454 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:32.609393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d35889-42c6-43f9-b8d7-368624a23dfa" (UID: "88d35889-42c6-43f9-b8d7-368624a23dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:32.710129 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:32.710089    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d35889-42c6-43f9-b8d7-368624a23dfa-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.365654 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.365624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5" event={"ID":"88d35889-42c6-43f9-b8d7-368624a23dfa","Type":"ContainerDied","Data":"d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a"}
Sep 10 23:54:33.365654 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.365651    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"
Sep 10 23:54:33.365876 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.365674    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d38a0bd15a4be531fec9025ea4c13da9154f30f0862a2a2e7d94abbdec41953a"
Sep 10 23:54:33.690196 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.690173    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:33.819422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819390    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819433    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jls52\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819449    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819464    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819537    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819583 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819960 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo\") pod \"dbdf9d81-360c-46a4-8f36-d7979c918535\" (UID: \"dbdf9d81-360c-46a4-8f36-d7979c918535\") "
Sep 10 23:54:33.819960 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819689    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:33.819960 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819717    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:33.819960 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:33.819960 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.819951    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:33.820186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.820056    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.820186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.820076    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.820186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.820089    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.820186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.820101    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.820186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.820051    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:33.821906 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.821855    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:33.821906 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.821880    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:33.822073 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.821954    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52" (OuterVolumeSpecName: "kube-api-access-jls52") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "kube-api-access-jls52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:33.822073 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.821965    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbdf9d81-360c-46a4-8f36-d7979c918535" (UID: "dbdf9d81-360c-46a4-8f36-d7979c918535"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 23:54:33.921169 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.921082    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.921169 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.921111    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdf9d81-360c-46a4-8f36-d7979c918535-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.921169 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.921119    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.921169 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.921129    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jls52\" (UniqueName: \"kubernetes.io/projected/dbdf9d81-360c-46a4-8f36-d7979c918535-kube-api-access-jls52\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:33.921169 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:33.921139    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdf9d81-360c-46a4-8f36-d7979c918535-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:34.246713 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.246635    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:34.371267 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.371229    2574 generic.go:356] "Generic (PLEG): container finished" podID="dbdf9d81-360c-46a4-8f36-d7979c918535" containerID="1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176" exitCode=0
Sep 10 23:54:34.371471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.371298    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"
Sep 10 23:54:34.371471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.371309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" event={"ID":"dbdf9d81-360c-46a4-8f36-d7979c918535","Type":"ContainerDied","Data":"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"}
Sep 10 23:54:34.371471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.371378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86" event={"ID":"dbdf9d81-360c-46a4-8f36-d7979c918535","Type":"ContainerDied","Data":"e32ae66bff656795cd9ecaffd2824691f9fd53868224a3f1f786106d6bd46c35"}
Sep 10 23:54:34.371471 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.371403    2574 scope.go:117] "RemoveContainer" containerID="1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"
Sep 10 23:54:34.381324 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.381305    2574 scope.go:117] "RemoveContainer" containerID="1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"
Sep 10 23:54:34.381581 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:34.381558    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176\": container with ID starting with 1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176 not found: ID does not exist" containerID="1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"
Sep 10 23:54:34.381645 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.381588    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176"} err="failed to get container status \"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176\": rpc error: code = NotFound desc = could not find container \"1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176\": container with ID starting with 1b3722f966200d73d1d0df9dabebd731f6bc7924c4bfb4fc788bd92d06b1e176 not found: ID does not exist"
Sep 10 23:54:34.395146 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.395065    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:34.397090 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.397066    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d43496-openshift-default-664b847b88-rdj86"]
Sep 10 23:54:34.615322 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:34.615286    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdf9d81-360c-46a4-8f36-d7979c918535" path="/var/lib/kubelet/pods/dbdf9d81-360c-46a4-8f36-d7979c918535/volumes"
Sep 10 23:54:40.239985 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:40.239933    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"]
Sep 10 23:54:40.241382 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:40.241360    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fe290-kv2h5"]
Sep 10 23:54:40.614152 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:40.614113    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d35889-42c6-43f9-b8d7-368624a23dfa" path="/var/lib/kubelet/pods/88d35889-42c6-43f9-b8d7-368624a23dfa/volumes"
Sep 10 23:54:43.362834 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.362793    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" podUID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" containerName="main" containerID="cri-o://e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a" gracePeriod=45
Sep 10 23:54:43.609817 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.609791    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:43.709657 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709588    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.709657 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709631    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.709657 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.709916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709679    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq2tl\" (UniqueName: \"kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.709916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.709916 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709792    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs\") pod \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\" (UID: \"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8\") "
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709947    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home" (OuterVolumeSpecName: "home") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.709934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.710048    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.710067    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:43.710083 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.710081    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:43.712019 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.711993    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:54:43.712250 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.712232    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm" (OuterVolumeSpecName: "dshm") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:54:43.712300 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.712258    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl" (OuterVolumeSpecName: "kube-api-access-mq2tl") pod "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" (UID: "da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8"). InnerVolumeSpecName "kube-api-access-mq2tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:43.810914 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.810884    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2tl\" (UniqueName: \"kubernetes.io/projected/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-kube-api-access-mq2tl\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:43.810914 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.810914    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:43.811119 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:43.810927    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:44.422383 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.422342    2574 generic.go:356] "Generic (PLEG): container finished" podID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" containerID="e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a" exitCode=0
Sep 10 23:54:44.422383 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.422384    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"
Sep 10 23:54:44.422891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.422434    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" event={"ID":"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8","Type":"ContainerDied","Data":"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"}
Sep 10 23:54:44.422891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.422476    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z" event={"ID":"da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8","Type":"ContainerDied","Data":"c2dc009e6fde299a6eb76d99249313e53aee3114b7b330d042f0fb2cc1f756c1"}
Sep 10 23:54:44.422891 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.422496    2574 scope.go:117] "RemoveContainer" containerID="e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"
Sep 10 23:54:44.432545 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.432530    2574 scope.go:117] "RemoveContainer" containerID="e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"
Sep 10 23:54:44.432822 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:44.432805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a\": container with ID starting with e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a not found: ID does not exist" containerID="e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"
Sep 10 23:54:44.432873 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.432834    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a"} err="failed to get container status \"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a\": rpc error: code = NotFound desc = could not find container \"e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a\": container with ID starting with e65f32df5d64a935a2eb0251799807fa49dbc0467c8756c9cae4c5def7861b8a not found: ID does not exist"
Sep 10 23:54:44.442947 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.442925    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:44.448553 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.448530    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d43496/noten-model-d43496-kserve-5bd75b69bc-nqq5z"]
Sep 10 23:54:44.614660 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:44.614629    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" path="/var/lib/kubelet/pods/da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8/volumes"
Sep 10 23:54:47.588181 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588145    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"]
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588506    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" containerName="main"
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588518    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" containerName="main"
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588536    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdf9d81-360c-46a4-8f36-d7979c918535" containerName="istio-proxy"
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588542    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdf9d81-360c-46a4-8f36-d7979c918535" containerName="istio-proxy"
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588552    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d35889-42c6-43f9-b8d7-368624a23dfa" containerName="revoke-keys"
Sep 10 23:54:47.588569 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588558    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d35889-42c6-43f9-b8d7-368624a23dfa" containerName="revoke-keys"
Sep 10 23:54:47.588761 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588611    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdf9d81-360c-46a4-8f36-d7979c918535" containerName="istio-proxy"
Sep 10 23:54:47.588761 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588622    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4dd248-f6f2-4d67-92f0-edfb5fa0f3d8" containerName="main"
Sep 10 23:54:47.588761 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.588629    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d35889-42c6-43f9-b8d7-368624a23dfa" containerName="revoke-keys"
Sep 10 23:54:47.592988 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.592967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:47.595298 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.595267    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:54:47.595549 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.595534    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:54:47.604232 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.604204    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"]
Sep 10 23:54:47.645339 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.645315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb\" (UID: \"8b9a3b3d-87be-417e-9a1b-56d562d17f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:47.746835 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.746804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb\" (UID: \"8b9a3b3d-87be-417e-9a1b-56d562d17f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:47.747425 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.747405    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb\" (UID: \"8b9a3b3d-87be-417e-9a1b-56d562d17f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:47.904220 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:47.904143    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:48.180318 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:48.180232    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"]
Sep 10 23:54:48.183071 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:54:48.183039    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9a3b3d_87be_417e_9a1b_56d562d17f87.slice/crio-117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a WatchSource:0}: Error finding container 117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a: Status 404 returned error can't find the container with id 117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a
Sep 10 23:54:48.445112 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:48.445084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb" event={"ID":"8b9a3b3d-87be-417e-9a1b-56d562d17f87","Type":"ContainerStarted","Data":"8b680f606b3b38105f805918cee58754ee271a2f655b823b207983bf15fe1c2b"}
Sep 10 23:54:48.445200 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:48.445133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb" event={"ID":"8b9a3b3d-87be-417e-9a1b-56d562d17f87","Type":"ContainerStarted","Data":"117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a"}
Sep 10 23:54:48.465822 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:48.465760    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb" podStartSLOduration=1.465745725 podStartE2EDuration="1.465745725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:48.464622761 +0000 UTC m=+1092.294449982" watchObservedRunningTime="2026-09-10 23:54:48.465745725 +0000 UTC m=+1092.295572944"
Sep 10 23:54:49.452384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:49.452346    2574 generic.go:356] "Generic (PLEG): container finished" podID="8b9a3b3d-87be-417e-9a1b-56d562d17f87" containerID="8b680f606b3b38105f805918cee58754ee271a2f655b823b207983bf15fe1c2b" exitCode=0
Sep 10 23:54:49.452753 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:49.452419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb" event={"ID":"8b9a3b3d-87be-417e-9a1b-56d562d17f87","Type":"ContainerDied","Data":"8b680f606b3b38105f805918cee58754ee271a2f655b823b207983bf15fe1c2b"}
Sep 10 23:54:50.587332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:50.587309    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:50.673938 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:50.673910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca\") pod \"8b9a3b3d-87be-417e-9a1b-56d562d17f87\" (UID: \"8b9a3b3d-87be-417e-9a1b-56d562d17f87\") "
Sep 10 23:54:50.674235 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:50.674214    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b9a3b3d-87be-417e-9a1b-56d562d17f87" (UID: "8b9a3b3d-87be-417e-9a1b-56d562d17f87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:50.775683 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:50.775607    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9a3b3d-87be-417e-9a1b-56d562d17f87-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:51.463550 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:51.463517    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"
Sep 10 23:54:51.463693 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:51.463515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb" event={"ID":"8b9a3b3d-87be-417e-9a1b-56d562d17f87","Type":"ContainerDied","Data":"117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a"}
Sep 10 23:54:51.463693 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:51.463646    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117aafb8468fbe0400e18d3d4fb23c0383b3da65709d45aa64c22d9bc479570a"
Sep 10 23:54:58.527990 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.527954    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:54:58.528943 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.528889    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" podUID="d1b9246a-ba44-427c-95f9-580dd96dfc47" containerName="payload-processing" containerID="cri-o://635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593" gracePeriod=30
Sep 10 23:54:58.538431 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.538404    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:54:58.538950 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.538674    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" podUID="d245709a-13fd-4882-a3b7-ad456916fcde" containerName="payload-pre-processing" containerID="cri-o://baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b" gracePeriod=30
Sep 10 23:54:58.700634 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.700593    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"]
Sep 10 23:54:58.703966 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.703942    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f598e-qgplb"]
Sep 10 23:54:58.784888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.784834    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:54:58.810275 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.810251    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:54:58.847701 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.847673    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume\") pod \"d1b9246a-ba44-427c-95f9-580dd96dfc47\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") "
Sep 10 23:54:58.847865 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.847834    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkzdm\" (UniqueName: \"kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm\") pod \"d1b9246a-ba44-427c-95f9-580dd96dfc47\" (UID: \"d1b9246a-ba44-427c-95f9-580dd96dfc47\") "
Sep 10 23:54:58.848109 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.848089    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1b9246a-ba44-427c-95f9-580dd96dfc47" (UID: "d1b9246a-ba44-427c-95f9-580dd96dfc47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:58.849964 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.849944    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm" (OuterVolumeSpecName: "kube-api-access-jkzdm") pod "d1b9246a-ba44-427c-95f9-580dd96dfc47" (UID: "d1b9246a-ba44-427c-95f9-580dd96dfc47"). InnerVolumeSpecName "kube-api-access-jkzdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:58.948917 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.948888    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfrdw\" (UniqueName: \"kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw\") pod \"d245709a-13fd-4882-a3b7-ad456916fcde\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") "
Sep 10 23:54:58.949103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.948926    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume\") pod \"d245709a-13fd-4882-a3b7-ad456916fcde\" (UID: \"d245709a-13fd-4882-a3b7-ad456916fcde\") "
Sep 10 23:54:58.949103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.949082    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzdm\" (UniqueName: \"kubernetes.io/projected/d1b9246a-ba44-427c-95f9-580dd96dfc47-kube-api-access-jkzdm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:58.949103 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.949094    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1b9246a-ba44-427c-95f9-580dd96dfc47-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:58.949384 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.949356    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d245709a-13fd-4882-a3b7-ad456916fcde" (UID: "d245709a-13fd-4882-a3b7-ad456916fcde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:54:58.950994 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:58.950963    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw" (OuterVolumeSpecName: "kube-api-access-zfrdw") pod "d245709a-13fd-4882-a3b7-ad456916fcde" (UID: "d245709a-13fd-4882-a3b7-ad456916fcde"). InnerVolumeSpecName "kube-api-access-zfrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:54:59.050149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.050075    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrdw\" (UniqueName: \"kubernetes.io/projected/d245709a-13fd-4882-a3b7-ad456916fcde-kube-api-access-zfrdw\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:59.050149 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.050104    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d245709a-13fd-4882-a3b7-ad456916fcde-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:54:59.504022 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.503988    2574 generic.go:356] "Generic (PLEG): container finished" podID="d245709a-13fd-4882-a3b7-ad456916fcde" containerID="baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b" exitCode=0
Sep 10 23:54:59.504219 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.504037    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"
Sep 10 23:54:59.504219 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.504078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" event={"ID":"d245709a-13fd-4882-a3b7-ad456916fcde","Type":"ContainerDied","Data":"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"}
Sep 10 23:54:59.504219 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.504132    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp" event={"ID":"d245709a-13fd-4882-a3b7-ad456916fcde","Type":"ContainerDied","Data":"14faedd4cca7871771fb1d0f11a878d96498096fe7562471a3fae059688bbeb3"}
Sep 10 23:54:59.504219 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.504160    2574 scope.go:117] "RemoveContainer" containerID="baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"
Sep 10 23:54:59.505582 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.505558    2574 generic.go:356] "Generic (PLEG): container finished" podID="d1b9246a-ba44-427c-95f9-580dd96dfc47" containerID="635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593" exitCode=0
Sep 10 23:54:59.505658 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.505592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" event={"ID":"d1b9246a-ba44-427c-95f9-580dd96dfc47","Type":"ContainerDied","Data":"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"}
Sep 10 23:54:59.505658 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.505606    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"
Sep 10 23:54:59.505658 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.505640    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v" event={"ID":"d1b9246a-ba44-427c-95f9-580dd96dfc47","Type":"ContainerDied","Data":"6f872f704c889b3052ed98b7353b8093e4b6911de228861fcc6bca1cbc4238cb"}
Sep 10 23:54:59.514437 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.514414    2574 scope.go:117] "RemoveContainer" containerID="baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"
Sep 10 23:54:59.514706 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:59.514685    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b\": container with ID starting with baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b not found: ID does not exist" containerID="baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"
Sep 10 23:54:59.514854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.514716    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b"} err="failed to get container status \"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b\": rpc error: code = NotFound desc = could not find container \"baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b\": container with ID starting with baad4a6f1b2c99607e3cbac345ca056e5d1209558a57d79096aaa1e08175a33b not found: ID does not exist"
Sep 10 23:54:59.514854 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.514739    2574 scope.go:117] "RemoveContainer" containerID="635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"
Sep 10 23:54:59.522709 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.522691    2574 scope.go:117] "RemoveContainer" containerID="635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"
Sep 10 23:54:59.523070 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:54:59.523046    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593\": container with ID starting with 635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593 not found: ID does not exist" containerID="635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"
Sep 10 23:54:59.523160 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.523082    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593"} err="failed to get container status \"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593\": rpc error: code = NotFound desc = could not find container \"635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593\": container with ID starting with 635c0a0ebad22624a2b1d76b07c3e7bb71fd3fcb9634a1332cc290d4866d9593 not found: ID does not exist"
Sep 10 23:54:59.528466 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.528443    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:54:59.535847 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.535823    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f598e-c8fcf4cd6-hh5hp"]
Sep 10 23:54:59.545961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.545938    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:54:59.549790 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:54:59.549754    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f598e-6f468495b7-lvj4v"]
Sep 10 23:55:00.613814 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:00.613750    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9a3b3d-87be-417e-9a1b-56d562d17f87" path="/var/lib/kubelet/pods/8b9a3b3d-87be-417e-9a1b-56d562d17f87/volumes"
Sep 10 23:55:00.614184 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:00.614144    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b9246a-ba44-427c-95f9-580dd96dfc47" path="/var/lib/kubelet/pods/d1b9246a-ba44-427c-95f9-580dd96dfc47/volumes"
Sep 10 23:55:00.614448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:00.614435    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d245709a-13fd-4882-a3b7-ad456916fcde" path="/var/lib/kubelet/pods/d245709a-13fd-4882-a3b7-ad456916fcde/volumes"
Sep 10 23:55:11.501026 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.500994    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501307    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b9246a-ba44-427c-95f9-580dd96dfc47" containerName="payload-processing"
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501318    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b9246a-ba44-427c-95f9-580dd96dfc47" containerName="payload-processing"
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501327    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d245709a-13fd-4882-a3b7-ad456916fcde" containerName="payload-pre-processing"
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501332    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d245709a-13fd-4882-a3b7-ad456916fcde" containerName="payload-pre-processing"
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501347    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9a3b3d-87be-417e-9a1b-56d562d17f87" containerName="revoke-keys"
Sep 10 23:55:11.501390 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501353    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9a3b3d-87be-417e-9a1b-56d562d17f87" containerName="revoke-keys"
Sep 10 23:55:11.501606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501417    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b9246a-ba44-427c-95f9-580dd96dfc47" containerName="payload-processing"
Sep 10 23:55:11.501606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501424    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9a3b3d-87be-417e-9a1b-56d562d17f87" containerName="revoke-keys"
Sep 10 23:55:11.501606 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.501431    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d245709a-13fd-4882-a3b7-ad456916fcde" containerName="payload-pre-processing"
Sep 10 23:55:11.504152 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.504123    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.506499 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.506474    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d7a539-dockercfg-mzcdp\""
Sep 10 23:55:11.506652 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.506503    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d7a539\""
Sep 10 23:55:11.522018 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.521993    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 10 23:55:11.661680 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.661639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6vsd\" (UniqueName: \"kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.661889 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.661706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.762662 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.762581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6vsd\" (UniqueName: \"kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.762662 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.762635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.763332 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.763313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.770691 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.770667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6vsd\" (UniqueName: \"kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd\") pod \"payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:11.816277 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:11.816241    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:12.087835 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:12.087721    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 10 23:55:12.090796 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:55:12.090754    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9d6d8e_941d_44d3_bce2_486b13053fa0.slice/crio-84214408bb2dd7cf5c3dbaf18f54bb686c245c20761dd37b608380db9434fda4 WatchSource:0}: Error finding container 84214408bb2dd7cf5c3dbaf18f54bb686c245c20761dd37b608380db9434fda4: Status 404 returned error can't find the container with id 84214408bb2dd7cf5c3dbaf18f54bb686c245c20761dd37b608380db9434fda4
Sep 10 23:55:12.571750 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:12.571658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" event={"ID":"1f9d6d8e-941d-44d3-bce2-486b13053fa0","Type":"ContainerStarted","Data":"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"}
Sep 10 23:55:12.571750 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:12.571717    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" event={"ID":"1f9d6d8e-941d-44d3-bce2-486b13053fa0","Type":"ContainerStarted","Data":"84214408bb2dd7cf5c3dbaf18f54bb686c245c20761dd37b608380db9434fda4"}
Sep 10 23:55:12.571750 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:12.571746    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:12.588613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:12.588551    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" podStartSLOduration=1.58853014 podStartE2EDuration="1.58853014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:12.587971208 +0000 UTC m=+1116.417798427" watchObservedRunningTime="2026-09-10 23:55:12.58853014 +0000 UTC m=+1116.418357360"
Sep 10 23:55:19.582359 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.582322    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 10 23:55:19.585038 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.585021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.587186 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.587157    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a78403-dockercfg-hz6nn\""
Sep 10 23:55:19.587300 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.587191    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a78403\""
Sep 10 23:55:19.596227 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.596207    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 10 23:55:19.737195 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.737163    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.737357 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.737204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tw9\" (UniqueName: \"kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.837839 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.837809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.838021 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.837850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tw9\" (UniqueName: \"kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.838591 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.838562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.846869 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.846840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tw9\" (UniqueName: \"kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9\") pod \"payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:19.895717 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:19.895690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:20.166731 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:20.166706    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 10 23:55:20.169878 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:55:20.169842    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db21074_d242_47ed_90ed_89e81b166af7.slice/crio-a0534fa1bca63e074c74fe32e9f806f89eb42a74b54123df62f199cc1faae301 WatchSource:0}: Error finding container a0534fa1bca63e074c74fe32e9f806f89eb42a74b54123df62f199cc1faae301: Status 404 returned error can't find the container with id a0534fa1bca63e074c74fe32e9f806f89eb42a74b54123df62f199cc1faae301
Sep 10 23:55:20.613695 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:20.613664    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:20.614075 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:20.613706    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" event={"ID":"5db21074-d242-47ed-90ed-89e81b166af7","Type":"ContainerStarted","Data":"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"}
Sep 10 23:55:20.614075 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:20.613725    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" event={"ID":"5db21074-d242-47ed-90ed-89e81b166af7","Type":"ContainerStarted","Data":"a0534fa1bca63e074c74fe32e9f806f89eb42a74b54123df62f199cc1faae301"}
Sep 10 23:55:20.646436 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:20.646383    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" podStartSLOduration=1.646360841 podStartE2EDuration="1.646360841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:20.642296225 +0000 UTC m=+1124.472123467" watchObservedRunningTime="2026-09-10 23:55:20.646360841 +0000 UTC m=+1124.476188061"
Sep 10 23:55:23.579152 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:23.579124    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 10 23:55:31.620961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:31.620930    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 10 23:55:41.708739 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.708706    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 10 23:55:41.713353 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.713334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.716098 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.716072    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4094ce-dockercfg-dpcpq\""
Sep 10 23:55:41.716281 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.716259    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4094ce\""
Sep 10 23:55:41.733910 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.733884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 10 23:55:41.836185 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.836155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.836359 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.836204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6jc\" (UniqueName: \"kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.937537 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.937501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.937722 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.937547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6jc\" (UniqueName: \"kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.938268 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.938247    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:41.946329 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:41.946304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6jc\" (UniqueName: \"kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc\") pod \"payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:42.023350 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.023256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:42.303347 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.303322    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 10 23:55:42.306111 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:55:42.306081    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4cf957_47f0_49bd_8410_d1c9f22e35da.slice/crio-d6ecb3809f48a3f71bcd8e6667c69e265604c15cf77dbb06cda119a9408c1d5d WatchSource:0}: Error finding container d6ecb3809f48a3f71bcd8e6667c69e265604c15cf77dbb06cda119a9408c1d5d: Status 404 returned error can't find the container with id d6ecb3809f48a3f71bcd8e6667c69e265604c15cf77dbb06cda119a9408c1d5d
Sep 10 23:55:42.732861 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.732821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" event={"ID":"4d4cf957-47f0-49bd-8410-d1c9f22e35da","Type":"ContainerStarted","Data":"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"}
Sep 10 23:55:42.733275 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.732981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:42.733275 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.733024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" event={"ID":"4d4cf957-47f0-49bd-8410-d1c9f22e35da","Type":"ContainerStarted","Data":"d6ecb3809f48a3f71bcd8e6667c69e265604c15cf77dbb06cda119a9408c1d5d"}
Sep 10 23:55:42.756836 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:42.756779    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" podStartSLOduration=1.7567446169999998 podStartE2EDuration="1.756744617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:42.755149621 +0000 UTC m=+1146.584976847" watchObservedRunningTime="2026-09-10 23:55:42.756744617 +0000 UTC m=+1146.586571839"
Sep 10 23:55:53.740106 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:53.740075    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 10 23:55:56.759363 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.759325    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:55:56.763980 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.763964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.767305 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.767282    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1f72544-dockercfg-2mnbx\""
Sep 10 23:55:56.767426 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.767335    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1f72544\""
Sep 10 23:55:56.775140 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.775109    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:55:56.799648 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.799619    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:55:56.803434 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.803419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:56.810723 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.810482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:55:56.866531 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.866499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ncx\" (UniqueName: \"kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.866682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.866543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.967442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.967402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ncx\" (UniqueName: \"kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.967442 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.967446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.967627 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.967488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:56.967627 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.967524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvzq\" (UniqueName: \"kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:56.968210 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.968190    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:56.975551 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:56.975519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ncx\" (UniqueName: \"kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx\") pod \"payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:57.068792 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.068710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.068912 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.068812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvzq\" (UniqueName: \"kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.069448 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.069424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.074405 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.074383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:57.076190 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.076166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvzq\" (UniqueName: \"kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq\") pod \"payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") " pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.114813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.114777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.345220 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.345144    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:55:57.347613 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:55:57.347589    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7494a7f1_728a_4e22_95d3_9d5f09fb8d26.slice/crio-ebda5701879c2e0069edc56ac60c7ec312551ff85daedb59c11dd549c7e0184b WatchSource:0}: Error finding container ebda5701879c2e0069edc56ac60c7ec312551ff85daedb59c11dd549c7e0184b: Status 404 returned error can't find the container with id ebda5701879c2e0069edc56ac60c7ec312551ff85daedb59c11dd549c7e0184b
Sep 10 23:55:57.401399 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.401347    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:55:57.403534 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:55:57.403509    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaade877a_72b5_4b62_8ab6_5912b851d0ac.slice/crio-aeea28422130e6e4df590d1bfb1d5d40c136dd5b8250a2b57b010be1583cc81d WatchSource:0}: Error finding container aeea28422130e6e4df590d1bfb1d5d40c136dd5b8250a2b57b010be1583cc81d: Status 404 returned error can't find the container with id aeea28422130e6e4df590d1bfb1d5d40c136dd5b8250a2b57b010be1583cc81d
Sep 10 23:55:57.819423 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.819333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" event={"ID":"7494a7f1-728a-4e22-95d3-9d5f09fb8d26","Type":"ContainerStarted","Data":"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"}
Sep 10 23:55:57.819423 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.819393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" event={"ID":"7494a7f1-728a-4e22-95d3-9d5f09fb8d26","Type":"ContainerStarted","Data":"ebda5701879c2e0069edc56ac60c7ec312551ff85daedb59c11dd549c7e0184b"}
Sep 10 23:55:57.819870 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.819501    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:55:57.820909 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.820882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" event={"ID":"aade877a-72b5-4b62-8ab6-5912b851d0ac","Type":"ContainerStarted","Data":"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"}
Sep 10 23:55:57.821055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.820929    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" event={"ID":"aade877a-72b5-4b62-8ab6-5912b851d0ac","Type":"ContainerStarted","Data":"aeea28422130e6e4df590d1bfb1d5d40c136dd5b8250a2b57b010be1583cc81d"}
Sep 10 23:55:57.821055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.820960    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:55:57.836620 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.836567    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" podStartSLOduration=1.83654743 podStartE2EDuration="1.83654743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:57.834263201 +0000 UTC m=+1161.664090421" watchObservedRunningTime="2026-09-10 23:55:57.83654743 +0000 UTC m=+1161.666374650"
Sep 10 23:55:57.850804 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:55:57.850752    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" podStartSLOduration=1.850740043 podStartE2EDuration="1.850740043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:57.849053102 +0000 UTC m=+1161.678880333" watchObservedRunningTime="2026-09-10 23:55:57.850740043 +0000 UTC m=+1161.680567263"
Sep 10 23:56:08.828613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:08.828537    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:56:08.828613 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:08.828603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:56:36.681120 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:36.681089    2574 scope.go:117] "RemoveContainer" containerID="6abeea0d89f1dbe2522e54d1ba6a7a9f758b305eddcb0c231871db12458db2bc"
Sep 10 23:56:37.328132 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.328091    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:56:37.333150 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.333129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.335373 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.335340    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"openshift-service-ca.crt\""
Sep 10 23:56:37.335499 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.335380    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"kube-root-ca.crt\""
Sep 10 23:56:37.336148 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.336124    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"auth-test-model-4094ce-kserve-self-signed-certs\""
Sep 10 23:56:37.336379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.336360    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"default-dockercfg-jn5c6\""
Sep 10 23:56:37.340048 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.340024    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:56:37.432447 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.432651 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.432651 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.432651 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ljz\" (UniqueName: \"kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.432822 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.432822 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.432730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533250 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533458 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533458 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ljz\" (UniqueName: \"kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533743 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533743 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.533886 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.533845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.535709 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.535682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.535974 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.535956    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.541323 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.541303    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ljz\" (UniqueName: \"kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz\") pod \"auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.645961 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.645917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:37.915069 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:37.914974    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:56:37.918292 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:56:37.918261    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05154ae9_b32d_423b_a096_8d7c66d5017b.slice/crio-d5c7c18497d7aaca24e2a42f8b15e900acba8917d09732fad72aeaa38bda10f6 WatchSource:0}: Error finding container d5c7c18497d7aaca24e2a42f8b15e900acba8917d09732fad72aeaa38bda10f6: Status 404 returned error can't find the container with id d5c7c18497d7aaca24e2a42f8b15e900acba8917d09732fad72aeaa38bda10f6
Sep 10 23:56:38.029682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:38.029496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" event={"ID":"05154ae9-b32d-423b-a096-8d7c66d5017b","Type":"ContainerStarted","Data":"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"}
Sep 10 23:56:38.029682 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:38.029681    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" event={"ID":"05154ae9-b32d-423b-a096-8d7c66d5017b","Type":"ContainerStarted","Data":"d5c7c18497d7aaca24e2a42f8b15e900acba8917d09732fad72aeaa38bda10f6"}
Sep 10 23:56:38.047562 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:38.047509    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" podStartSLOduration=1.047492479 podStartE2EDuration="1.047492479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:38.046825501 +0000 UTC m=+1201.876652721" watchObservedRunningTime="2026-09-10 23:56:38.047492479 +0000 UTC m=+1201.877319697"
Sep 10 23:56:47.647042 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:47.647003    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:47.649488 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:47.647065    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:47.659621 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:47.659601    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:48.092201 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:48.092170    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:56:59.590845 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.590807    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"]
Sep 10 23:56:59.594445 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.594430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:56:59.596446 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.596422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:56:59.596446 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.596427    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:56:59.601935 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.601910    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"]
Sep 10 23:56:59.631352 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.631322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb\" (UID: \"7b75774b-b42a-4b56-b891-e70955a61718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:56:59.732485 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.732450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb\" (UID: \"7b75774b-b42a-4b56-b891-e70955a61718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:56:59.733139 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.733114    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb\" (UID: \"7b75774b-b42a-4b56-b891-e70955a61718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:56:59.906837 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:56:59.906748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:57:00.177879 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:00.177852    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"]
Sep 10 23:57:00.180408 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:57:00.180384    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b75774b_b42a_4b56_b891_e70955a61718.slice/crio-0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245 WatchSource:0}: Error finding container 0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245: Status 404 returned error can't find the container with id 0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245
Sep 10 23:57:01.149237 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:01.149195    2574 generic.go:356] "Generic (PLEG): container finished" podID="7b75774b-b42a-4b56-b891-e70955a61718" containerID="6e59998265cc134712cda7d9993d3ff76aa34fe45786744c29011a9dadc9a09d" exitCode=0
Sep 10 23:57:01.149237 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:01.149240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb" event={"ID":"7b75774b-b42a-4b56-b891-e70955a61718","Type":"ContainerDied","Data":"6e59998265cc134712cda7d9993d3ff76aa34fe45786744c29011a9dadc9a09d"}
Sep 10 23:57:01.149955 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:01.149277    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb" event={"ID":"7b75774b-b42a-4b56-b891-e70955a61718","Type":"ContainerStarted","Data":"0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245"}
Sep 10 23:57:02.287630 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:02.287610    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:57:02.356542 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:02.356516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca\") pod \"7b75774b-b42a-4b56-b891-e70955a61718\" (UID: \"7b75774b-b42a-4b56-b891-e70955a61718\") "
Sep 10 23:57:02.356888 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:02.356866    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b75774b-b42a-4b56-b891-e70955a61718" (UID: "7b75774b-b42a-4b56-b891-e70955a61718"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:02.457618 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:02.457551    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75774b-b42a-4b56-b891-e70955a61718-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:57:03.160752 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:03.160715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb" event={"ID":"7b75774b-b42a-4b56-b891-e70955a61718","Type":"ContainerDied","Data":"0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245"}
Sep 10 23:57:03.160752 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:03.160747    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"
Sep 10 23:57:03.160983 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:03.160789    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca1cc008ee93388dcd4adbdf75cb19e4e2443b6dc20a4a20615bbcef2a58245"
Sep 10 23:57:15.172360 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.172327    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:57:15.172927 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.172558    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" podUID="aade877a-72b5-4b62-8ab6-5912b851d0ac" containerName="payload-processing" containerID="cri-o://89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1" gracePeriod=30
Sep 10 23:57:15.180174 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.180142    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:57:15.180394 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.180338    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" podUID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" containerName="payload-pre-processing" containerID="cri-o://fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee" gracePeriod=30
Sep 10 23:57:15.360670 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.360597    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"]
Sep 10 23:57:15.362519 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.362492    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1f72544-5t5tb"]
Sep 10 23:57:15.449957 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.449928    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:57:15.452610 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.452591    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:57:15.571703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.571670    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume\") pod \"aade877a-72b5-4b62-8ab6-5912b851d0ac\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") "
Sep 10 23:57:15.571703 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.571713    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jvzq\" (UniqueName: \"kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq\") pod \"aade877a-72b5-4b62-8ab6-5912b851d0ac\" (UID: \"aade877a-72b5-4b62-8ab6-5912b851d0ac\") "
Sep 10 23:57:15.571949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.571744    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume\") pod \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") "
Sep 10 23:57:15.571949 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.571834    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9ncx\" (UniqueName: \"kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx\") pod \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\" (UID: \"7494a7f1-728a-4e22-95d3-9d5f09fb8d26\") "
Sep 10 23:57:15.572192 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.572160    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7494a7f1-728a-4e22-95d3-9d5f09fb8d26" (UID: "7494a7f1-728a-4e22-95d3-9d5f09fb8d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:15.572192 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.572182    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aade877a-72b5-4b62-8ab6-5912b851d0ac" (UID: "aade877a-72b5-4b62-8ab6-5912b851d0ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:57:15.573809 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.573780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq" (OuterVolumeSpecName: "kube-api-access-8jvzq") pod "aade877a-72b5-4b62-8ab6-5912b851d0ac" (UID: "aade877a-72b5-4b62-8ab6-5912b851d0ac"). InnerVolumeSpecName "kube-api-access-8jvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:15.573957 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.573934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx" (OuterVolumeSpecName: "kube-api-access-m9ncx") pod "7494a7f1-728a-4e22-95d3-9d5f09fb8d26" (UID: "7494a7f1-728a-4e22-95d3-9d5f09fb8d26"). InnerVolumeSpecName "kube-api-access-m9ncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:57:15.672894 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.672866    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aade877a-72b5-4b62-8ab6-5912b851d0ac-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.672894 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.672893    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvzq\" (UniqueName: \"kubernetes.io/projected/aade877a-72b5-4b62-8ab6-5912b851d0ac-kube-api-access-8jvzq\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.673069 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.672902    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:57:15.673069 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:15.672911    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ncx\" (UniqueName: \"kubernetes.io/projected/7494a7f1-728a-4e22-95d3-9d5f09fb8d26-kube-api-access-m9ncx\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:57:16.228005 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.227967    2574 generic.go:356] "Generic (PLEG): container finished" podID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" containerID="fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee" exitCode=0
Sep 10 23:57:16.228418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.228024    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"
Sep 10 23:57:16.228418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.228053    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" event={"ID":"7494a7f1-728a-4e22-95d3-9d5f09fb8d26","Type":"ContainerDied","Data":"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"}
Sep 10 23:57:16.228418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.228111    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8" event={"ID":"7494a7f1-728a-4e22-95d3-9d5f09fb8d26","Type":"ContainerDied","Data":"ebda5701879c2e0069edc56ac60c7ec312551ff85daedb59c11dd549c7e0184b"}
Sep 10 23:57:16.228418 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.228139    2574 scope.go:117] "RemoveContainer" containerID="fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"
Sep 10 23:57:16.229842 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.229818    2574 generic.go:356] "Generic (PLEG): container finished" podID="aade877a-72b5-4b62-8ab6-5912b851d0ac" containerID="89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1" exitCode=0
Sep 10 23:57:16.229956 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.229929    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"
Sep 10 23:57:16.230035 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.229856    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" event={"ID":"aade877a-72b5-4b62-8ab6-5912b851d0ac","Type":"ContainerDied","Data":"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"}
Sep 10 23:57:16.230165 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.230070    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9" event={"ID":"aade877a-72b5-4b62-8ab6-5912b851d0ac","Type":"ContainerDied","Data":"aeea28422130e6e4df590d1bfb1d5d40c136dd5b8250a2b57b010be1583cc81d"}
Sep 10 23:57:16.237507 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.237487    2574 scope.go:117] "RemoveContainer" containerID="fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"
Sep 10 23:57:16.237951 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:57:16.237925    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee\": container with ID starting with fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee not found: ID does not exist" containerID="fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"
Sep 10 23:57:16.238040 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.237961    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee"} err="failed to get container status \"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee\": rpc error: code = NotFound desc = could not find container \"fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee\": container with ID starting with fa29cac4f466c88e950c0d50c87fc8b2f67cf3b503dfb256c6a3b7899fee58ee not found: ID does not exist"
Sep 10 23:57:16.238040 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.237984    2574 scope.go:117] "RemoveContainer" containerID="89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"
Sep 10 23:57:16.246241 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.246221    2574 scope.go:117] "RemoveContainer" containerID="89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"
Sep 10 23:57:16.246504 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:57:16.246486    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1\": container with ID starting with 89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1 not found: ID does not exist" containerID="89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"
Sep 10 23:57:16.246559 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.246511    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1"} err="failed to get container status \"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1\": rpc error: code = NotFound desc = could not find container \"89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1\": container with ID starting with 89fbdcf502ef83a79f1f3c228be724691802e6f10cb2bd983c710dd9c42851d1 not found: ID does not exist"
Sep 10 23:57:16.256982 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.256961    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:57:16.258547 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.258530    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1f72544-556b6f84fc-qcnx9"]
Sep 10 23:57:16.268270 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.268250    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:57:16.274435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.274414    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1f72544-c6c79dcdb-fg4q8"]
Sep 10 23:57:16.614750 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.614717    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" path="/var/lib/kubelet/pods/7494a7f1-728a-4e22-95d3-9d5f09fb8d26/volumes"
Sep 10 23:57:16.615075 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.615062    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b75774b-b42a-4b56-b891-e70955a61718" path="/var/lib/kubelet/pods/7b75774b-b42a-4b56-b891-e70955a61718/volumes"
Sep 10 23:57:16.615388 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:16.615376    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aade877a-72b5-4b62-8ab6-5912b851d0ac" path="/var/lib/kubelet/pods/aade877a-72b5-4b62-8ab6-5912b851d0ac/volumes"
Sep 10 23:57:36.700357 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:36.700288    2574 scope.go:117] "RemoveContainer" containerID="e05ae244daed0f55c8ac2510f2b7ae0297331953c7f51c8128c0066c9e76869e"
Sep 10 23:57:36.708283 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:36.708263    2574 scope.go:117] "RemoveContainer" containerID="f0092949041af49ee178a0f7fe9d39ab15ec76055c66ebb08996b5a790b8842d"
Sep 10 23:57:36.715479 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:57:36.715462    2574 scope.go:117] "RemoveContainer" containerID="462aac5461dbc3b873906a223584131302a37b42829dd56b18ead9ff99a097c8"
Sep 10 23:58:36.747743 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:36.747713    2574 scope.go:117] "RemoveContainer" containerID="39fec19cda335936c2851c3db68a8a20ebe7c23fd0cdaf1f7711e50d0af3ed4c"
Sep 10 23:58:37.903379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:37.903344    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:58:52.922338 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:52.922266    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" podUID="05154ae9-b32d-423b-a096-8d7c66d5017b" containerName="main" containerID="cri-o://b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3" gracePeriod=45
Sep 10 23:58:53.172909 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.172850    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:58:53.246153 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246122    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246213    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2ljz\" (UniqueName: \"kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246238    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246309 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246524 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246312    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home\") pod \"05154ae9-b32d-423b-a096-8d7c66d5017b\" (UID: \"05154ae9-b32d-423b-a096-8d7c66d5017b\") "
Sep 10 23:58:53.246587 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246513    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:53.246643 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246621    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache" (OuterVolumeSpecName: "model-cache") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:53.246685 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.246644    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home" (OuterVolumeSpecName: "home") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:53.248529 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.248502    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm" (OuterVolumeSpecName: "dshm") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:58:53.248641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.248530    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:58:53.248641 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.248533    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz" (OuterVolumeSpecName: "kube-api-access-k2ljz") pod "05154ae9-b32d-423b-a096-8d7c66d5017b" (UID: "05154ae9-b32d-423b-a096-8d7c66d5017b"). InnerVolumeSpecName "kube-api-access-k2ljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:58:53.347624 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347599    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05154ae9-b32d-423b-a096-8d7c66d5017b-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.347624 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347624    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ljz\" (UniqueName: \"kubernetes.io/projected/05154ae9-b32d-423b-a096-8d7c66d5017b-kube-api-access-k2ljz\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.347801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347634    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.347801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347642    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.347801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347649    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.347801 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.347657    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05154ae9-b32d-423b-a096-8d7c66d5017b-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:58:53.721085 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.721046    2574 generic.go:356] "Generic (PLEG): container finished" podID="05154ae9-b32d-423b-a096-8d7c66d5017b" containerID="b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3" exitCode=0
Sep 10 23:58:53.721280 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.721092    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"
Sep 10 23:58:53.721280 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.721131    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" event={"ID":"05154ae9-b32d-423b-a096-8d7c66d5017b","Type":"ContainerDied","Data":"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"}
Sep 10 23:58:53.721280 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.721183    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl" event={"ID":"05154ae9-b32d-423b-a096-8d7c66d5017b","Type":"ContainerDied","Data":"d5c7c18497d7aaca24e2a42f8b15e900acba8917d09732fad72aeaa38bda10f6"}
Sep 10 23:58:53.721280 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.721203    2574 scope.go:117] "RemoveContainer" containerID="b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"
Sep 10 23:58:53.731729 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.731710    2574 scope.go:117] "RemoveContainer" containerID="b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"
Sep 10 23:58:53.732036 ip-10-0-131-31 kubenswrapper[2574]: E0910 23:58:53.732013    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3\": container with ID starting with b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3 not found: ID does not exist" containerID="b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"
Sep 10 23:58:53.732113 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.732044    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3"} err="failed to get container status \"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3\": rpc error: code = NotFound desc = could not find container \"b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3\": container with ID starting with b207a4a974caa2652f6e845bc69dd2d209d742fe573fc210b451f2bbb4e52da3 not found: ID does not exist"
Sep 10 23:58:53.742192 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.742169    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:58:53.745487 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:53.745466    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/auth-test-model-4094ce-kserve-5446d5bbd7-gfwwl"]
Sep 10 23:58:54.614519 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:58:54.614485    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05154ae9-b32d-423b-a096-8d7c66d5017b" path="/var/lib/kubelet/pods/05154ae9-b32d-423b-a096-8d7c66d5017b/volumes"
Sep 10 23:59:03.739674 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739588    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739952    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aade877a-72b5-4b62-8ab6-5912b851d0ac" containerName="payload-processing"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739964    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aade877a-72b5-4b62-8ab6-5912b851d0ac" containerName="payload-processing"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739978    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05154ae9-b32d-423b-a096-8d7c66d5017b" containerName="main"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739983    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="05154ae9-b32d-423b-a096-8d7c66d5017b" containerName="main"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739991    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" containerName="payload-pre-processing"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.739997    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" containerName="payload-pre-processing"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740009    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b75774b-b42a-4b56-b891-e70955a61718" containerName="revoke-keys"
Sep 10 23:59:03.740055 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740016    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b75774b-b42a-4b56-b891-e70955a61718" containerName="revoke-keys"
Sep 10 23:59:03.740345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740082    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aade877a-72b5-4b62-8ab6-5912b851d0ac" containerName="payload-processing"
Sep 10 23:59:03.740345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740093    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7494a7f1-728a-4e22-95d3-9d5f09fb8d26" containerName="payload-pre-processing"
Sep 10 23:59:03.740345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740101    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="05154ae9-b32d-423b-a096-8d7c66d5017b" containerName="main"
Sep 10 23:59:03.740345 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.740107    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b75774b-b42a-4b56-b891-e70955a61718" containerName="revoke-keys"
Sep 10 23:59:03.743100 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.743084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.745184 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.745162    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"openshift-service-ca.crt\""
Sep 10 23:59:03.746082 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.746047    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"kube-root-ca.crt\""
Sep 10 23:59:03.746082 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.746067    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"default-dockercfg-2scc9\""
Sep 10 23:59:03.746233 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.746061    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d7a539\"/\"sub-test-model-d7a539-kserve-self-signed-certs\""
Sep 10 23:59:03.751388 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.751365    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 10 23:59:03.841352 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjqp\" (UniqueName: \"kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.841525 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.841525 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.841525 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.841525 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841512    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.841698 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.841547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942297 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjqp\" (UniqueName: \"kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942478 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942478 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942478 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942478 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942478 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942816 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942900 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.942900 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.942890    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.944584 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.944557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.944932 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.944917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:03.952306 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:03.952282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjqp\" (UniqueName: \"kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp\") pod \"sub-test-model-d7a539-kserve-858d4cd4b5-phrcc\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") " pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:04.054052 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:04.053971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:04.324826 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:04.324795    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 10 23:59:04.326448 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:59:04.326413    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0959f228_384a_4314_9f56_b5a3d814a67c.slice/crio-ec775513334764f44b11224c5e68bf37ad61aa6723bbd36804d657d46024f443 WatchSource:0}: Error finding container ec775513334764f44b11224c5e68bf37ad61aa6723bbd36804d657d46024f443: Status 404 returned error can't find the container with id ec775513334764f44b11224c5e68bf37ad61aa6723bbd36804d657d46024f443
Sep 10 23:59:04.779213 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:04.779168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" event={"ID":"0959f228-384a-4314-9f56-b5a3d814a67c","Type":"ContainerStarted","Data":"28cb2b379488641986544fb12aee2b4f65a768e2397e6c6a66d963c3666bd0a9"}
Sep 10 23:59:04.779213 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:04.779214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" event={"ID":"0959f228-384a-4314-9f56-b5a3d814a67c","Type":"ContainerStarted","Data":"ec775513334764f44b11224c5e68bf37ad61aa6723bbd36804d657d46024f443"}
Sep 10 23:59:04.794455 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:04.794410    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" podStartSLOduration=1.794394281 podStartE2EDuration="1.794394281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:04.79334356 +0000 UTC m=+1348.623170778" watchObservedRunningTime="2026-09-10 23:59:04.794394281 +0000 UTC m=+1348.624221499"
Sep 10 23:59:14.054813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.054756    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:14.054813 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.054822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:14.067204 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.067167    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:14.846572 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.846542    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 10 23:59:14.962793 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.962749    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 10 23:59:14.965793 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.965754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:14.967929 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.967913    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4ec4c8\""
Sep 10 23:59:14.978046 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:14.978026    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 10 23:59:15.001438 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.001416    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 10 23:59:15.004873 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.004857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.007568 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.007550    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4ec4c8-dockercfg-9c87j\""
Sep 10 23:59:15.013685 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.013665    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 10 23:59:15.142417 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.142315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.142838 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.142409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmb6\" (UniqueName: \"kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.142838 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.142557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mfzr\" (UniqueName: \"kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.142838 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.142600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.243224 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.243397 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.243397 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmb6\" (UniqueName: \"kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.243493 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfzr\" (UniqueName: \"kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.243984 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.244048 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.243968    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.251049 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.251028    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmb6\" (UniqueName: \"kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6\") pod \"payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.251164 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.251099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mfzr\" (UniqueName: \"kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr\") pod \"payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.277038 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.277016    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.314676 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.314647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.547818 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.547793    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 10 23:59:15.549574 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:59:15.549547    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb02eced_7e3f_4b71_a89f_1dc1a90e13c9.slice/crio-8e51a29da7bac4f568a6e0d20ff72c5e17bc67c7ae25073b7818b51ff9494109 WatchSource:0}: Error finding container 8e51a29da7bac4f568a6e0d20ff72c5e17bc67c7ae25073b7818b51ff9494109: Status 404 returned error can't find the container with id 8e51a29da7bac4f568a6e0d20ff72c5e17bc67c7ae25073b7818b51ff9494109
Sep 10 23:59:15.595387 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.595361    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 10 23:59:15.596982 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:59:15.596957    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd4b7b_4a77_4929_8114_920fd618a738.slice/crio-4b767a8f78d82aa1ace77ffe75290232b25e3ff552580a3b744354dff1e8c373 WatchSource:0}: Error finding container 4b767a8f78d82aa1ace77ffe75290232b25e3ff552580a3b744354dff1e8c373: Status 404 returned error can't find the container with id 4b767a8f78d82aa1ace77ffe75290232b25e3ff552580a3b744354dff1e8c373
Sep 10 23:59:15.845896 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.845857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" event={"ID":"9bcd4b7b-4a77-4929-8114-920fd618a738","Type":"ContainerStarted","Data":"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"}
Sep 10 23:59:15.846077 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.845942    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" event={"ID":"9bcd4b7b-4a77-4929-8114-920fd618a738","Type":"ContainerStarted","Data":"4b767a8f78d82aa1ace77ffe75290232b25e3ff552580a3b744354dff1e8c373"}
Sep 10 23:59:15.846077 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.846052    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:15.847677 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.847652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" event={"ID":"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9","Type":"ContainerStarted","Data":"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"}
Sep 10 23:59:15.847817 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.847692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" event={"ID":"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9","Type":"ContainerStarted","Data":"8e51a29da7bac4f568a6e0d20ff72c5e17bc67c7ae25073b7818b51ff9494109"}
Sep 10 23:59:15.847817 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.847731    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:15.864396 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.864342    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" podStartSLOduration=1.864329911 podStartE2EDuration="1.864329911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:15.862043736 +0000 UTC m=+1359.691870959" watchObservedRunningTime="2026-09-10 23:59:15.864329911 +0000 UTC m=+1359.694157129"
Sep 10 23:59:15.879683 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:15.879639    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" podStartSLOduration=1.879628019 podStartE2EDuration="1.879628019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:15.877671267 +0000 UTC m=+1359.707498488" watchObservedRunningTime="2026-09-10 23:59:15.879628019 +0000 UTC m=+1359.709455238"
Sep 10 23:59:26.854719 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:26.854688    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 10 23:59:26.855176 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:26.854849    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 10 23:59:37.237970 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.237929    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 10 23:59:37.241575 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.241557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.245127 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.245107    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"default-dockercfg-jn5c6\""
Sep 10 23:59:37.245665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.245642    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"kube-root-ca.crt\""
Sep 10 23:59:37.245665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.245659    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"sub-test-model-4094ce-kserve-self-signed-certs\""
Sep 10 23:59:37.245853 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.245685    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4094ce\"/\"openshift-service-ca.crt\""
Sep 10 23:59:37.251456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.251438    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 10 23:59:37.333493 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.333629 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.333629 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.333629 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.333785 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdrm\" (UniqueName: \"kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.333785 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.333692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.434922 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.434891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.434935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.434960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdrm\" (UniqueName: \"kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435325 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435311    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435435 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435491 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.435491 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.435449    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.437262 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.437242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.437386 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.437369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.443456 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.443434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdrm\" (UniqueName: \"kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm\") pod \"sub-test-model-4094ce-kserve-598d77d8cc-zxh2l\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") " pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.551413 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.551332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:37.825379 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.825339    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 10 23:59:37.827192 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:59:37.827159    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dc169ab_92c7_4dbf_933e_3eadc64b2adf.slice/crio-682e9d3d1c917d5ccf50ff368179613f2dd9209a72151830603d7af8dba41e3b WatchSource:0}: Error finding container 682e9d3d1c917d5ccf50ff368179613f2dd9209a72151830603d7af8dba41e3b: Status 404 returned error can't find the container with id 682e9d3d1c917d5ccf50ff368179613f2dd9209a72151830603d7af8dba41e3b
Sep 10 23:59:37.958306 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.958266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" event={"ID":"4dc169ab-92c7-4dbf-933e-3eadc64b2adf","Type":"ContainerStarted","Data":"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"}
Sep 10 23:59:37.958458 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.958314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" event={"ID":"4dc169ab-92c7-4dbf-933e-3eadc64b2adf","Type":"ContainerStarted","Data":"682e9d3d1c917d5ccf50ff368179613f2dd9209a72151830603d7af8dba41e3b"}
Sep 10 23:59:37.976500 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:37.976459    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" podStartSLOduration=0.976444921 podStartE2EDuration="976.444921ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:37.974665678 +0000 UTC m=+1381.804492900" watchObservedRunningTime="2026-09-10 23:59:37.976444921 +0000 UTC m=+1381.806272175"
Sep 10 23:59:41.083051 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.083015    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"]
Sep 10 23:59:41.086354 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.086331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:41.088285 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.088265    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 10 23:59:41.088429 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.088412    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:59:41.094814 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.094786    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"]
Sep 10 23:59:41.170302 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.170268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv\" (UID: \"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:41.271070 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.271037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv\" (UID: \"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:41.271665 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.271641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv\" (UID: \"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:41.406655 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.406627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:41.678706 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.678590    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"]
Sep 10 23:59:41.680825 ip-10-0-131-31 kubenswrapper[2574]: W0910 23:59:41.680796    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a0515c_4f12_4bf9_afad_43f4a1c2fa98.slice/crio-052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742 WatchSource:0}: Error finding container 052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742: Status 404 returned error can't find the container with id 052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742
Sep 10 23:59:41.979878 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.979851    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv" event={"ID":"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98","Type":"ContainerStarted","Data":"fafdbc55d9c29cae24d95f07c76ff3c39dc8e49d93875c12bfdaa6b11ccb3d24"}
Sep 10 23:59:41.980002 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.979900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv" event={"ID":"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98","Type":"ContainerStarted","Data":"052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742"}
Sep 10 23:59:41.995589 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:41.995540    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv" podStartSLOduration=0.995524921 podStartE2EDuration="995.524921ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:41.994465643 +0000 UTC m=+1385.824292864" watchObservedRunningTime="2026-09-10 23:59:41.995524921 +0000 UTC m=+1385.825352140"
Sep 10 23:59:42.986661 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:42.986619    2574 generic.go:356] "Generic (PLEG): container finished" podID="a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" containerID="fafdbc55d9c29cae24d95f07c76ff3c39dc8e49d93875c12bfdaa6b11ccb3d24" exitCode=0
Sep 10 23:59:42.987064 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:42.986698    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv" event={"ID":"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98","Type":"ContainerDied","Data":"fafdbc55d9c29cae24d95f07c76ff3c39dc8e49d93875c12bfdaa6b11ccb3d24"}
Sep 10 23:59:44.124576 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.124553    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:44.197782 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.197730    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca\") pod \"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98\" (UID: \"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98\") "
Sep 10 23:59:44.198104 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.198077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" (UID: "a6a0515c-4f12-4bf9-afad-43f4a1c2fa98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:59:44.299560 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.299490    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 10 23:59:44.997422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.997341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv" event={"ID":"a6a0515c-4f12-4bf9-afad-43f4a1c2fa98","Type":"ContainerDied","Data":"052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742"}
Sep 10 23:59:44.997422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.997395    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052b8a7cc42378cb297130369af2a9d96fbc7b4b673715e50681aed20fc8e742"
Sep 10 23:59:44.997422 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:44.997358    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"
Sep 10 23:59:47.551473 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:47.551437    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:47.551890 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:47.551487    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:47.564635 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:47.564604    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:48.023958 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:48.023927    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 10 23:59:56.758251 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:56.758213    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"]
Sep 10 23:59:56.761087 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:56.761056    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8b47cf-h6zxv"]
Sep 10 23:59:58.615108 ip-10-0-131-31 kubenswrapper[2574]: I0910 23:59:58.615076    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" path="/var/lib/kubelet/pods/a6a0515c-4f12-4bf9-afad-43f4a1c2fa98/volumes"
Sep 11 00:00:00.153252 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.153222    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"]
Sep 11 00:00:00.153661 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.153628    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" containerName="revoke-keys"
Sep 11 00:00:00.153661 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.153639    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" containerName="revoke-keys"
Sep 11 00:00:00.153747 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.153696    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a0515c-4f12-4bf9-afad-43f4a1c2fa98" containerName="revoke-keys"
Sep 11 00:00:00.157729 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.157710    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"]
Sep 11 00:00:00.157872 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.157850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"
Sep 11 00:00:00.160925 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.160909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"
Sep 11 00:00:00.161901 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.161881    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:00:00.168357 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.168335    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"]
Sep 11 00:00:00.171840 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.171822    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"
Sep 11 00:00:00.177840 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.177479    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"]
Sep 11 00:00:00.179593 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.179569    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"]
Sep 11 00:00:00.196747 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.196726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"
Sep 11 00:00:00.206400 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.206201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"
Sep 11 00:00:00.210478 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.210273    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"]
Sep 11 00:00:00.214794 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.214755    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"]
Sep 11 00:00:00.219096 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.219060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"
Sep 11 00:00:00.239241 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.239150    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"]
Sep 11 00:00:00.259536 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.254727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"
Sep 11 00:00:00.262845 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.262801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"
Sep 11 00:00:00.503691 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.503655    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"]
Sep 11 00:00:00.507831 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:00.505355    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a93e703_9f16_41ae_a5de_89b9723b211a.slice/crio-51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc WatchSource:0}: Error finding container 51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc: Status 404 returned error can't find the container with id 51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc
Sep 11 00:00:00.529181 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.529151    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"]
Sep 11 00:00:00.536893 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:00.536861    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee755d0c_6fdb_4fd4_bcd5_ff458f1c54cb.slice/crio-af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f WatchSource:0}: Error finding container af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f: Status 404 returned error can't find the container with id af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f
Sep 11 00:00:00.574545 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.574519    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"]
Sep 11 00:00:00.577492 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:00.577464    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdc979f_ab57_49f9_9d20_7b51be5733b4.slice/crio-7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a WatchSource:0}: Error finding container 7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a: Status 404 returned error can't find the container with id 7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a
Sep 11 00:00:00.578428 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:00.578407    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"]
Sep 11 00:00:00.581132 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:00.581112    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270bda53_cb44_489c_baaa_ca6d6b7ff250.slice/crio-68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e WatchSource:0}: Error finding container 68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e: Status 404 returned error can't find the container with id 68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e
Sep 11 00:00:01.078483 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.078385    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt" event={"ID":"270bda53-cb44-489c-baaa-ca6d6b7ff250","Type":"ContainerStarted","Data":"7dfa26439d4d4bfb95541bb28ee75b7ba61d19ba9758f1e300c339b4dbf42c49"}
Sep 11 00:00:01.078483 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.078449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt" event={"ID":"270bda53-cb44-489c-baaa-ca6d6b7ff250","Type":"ContainerStarted","Data":"68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e"}
Sep 11 00:00:01.079936 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.079907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r" event={"ID":"4bdc979f-ab57-49f9-9d20-7b51be5733b4","Type":"ContainerStarted","Data":"614b0734d086da9f27689ddf77b54df1910e388651422f11e2c10bb67e4e42b2"}
Sep 11 00:00:01.080054 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.079954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r" event={"ID":"4bdc979f-ab57-49f9-9d20-7b51be5733b4","Type":"ContainerStarted","Data":"7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a"}
Sep 11 00:00:01.081445 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.081416    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f" event={"ID":"ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb","Type":"ContainerStarted","Data":"d0ee3c26cb8d996e7a6717fd2e4112b148b5f74068ebbe83ab9531b87a7af2ca"}
Sep 11 00:00:01.081566 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.081499    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f" event={"ID":"ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb","Type":"ContainerStarted","Data":"af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f"}
Sep 11 00:00:01.082895 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.082869    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w" event={"ID":"9a93e703-9f16-41ae-a5de-89b9723b211a","Type":"ContainerStarted","Data":"cf40454875f8269cf215c696f6fb8503f04c83892ad30d5b77f76dc2f145891f"}
Sep 11 00:00:01.082970 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.082907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w" event={"ID":"9a93e703-9f16-41ae-a5de-89b9723b211a","Type":"ContainerStarted","Data":"51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc"}
Sep 11 00:00:01.095120 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.095079    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt" podStartSLOduration=1.095066742 podStartE2EDuration="1.095066742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.092643862 +0000 UTC m=+1404.922471081" watchObservedRunningTime="2026-09-11 00:00:01.095066742 +0000 UTC m=+1404.924893961"
Sep 11 00:00:01.109112 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.109062    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r" podStartSLOduration=1.109048646 podStartE2EDuration="1.109048646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.107427019 +0000 UTC m=+1404.937254240" watchObservedRunningTime="2026-09-11 00:00:01.109048646 +0000 UTC m=+1404.938875865"
Sep 11 00:00:01.126017 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.125971    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w" podStartSLOduration=1.125957572 podStartE2EDuration="1.125957572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.123811619 +0000 UTC m=+1404.953638836" watchObservedRunningTime="2026-09-11 00:00:01.125957572 +0000 UTC m=+1404.955784792"
Sep 11 00:00:01.140868 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:01.140812    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f" podStartSLOduration=1.140798593 podStartE2EDuration="1.140798593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.138326416 +0000 UTC m=+1404.968153649" watchObservedRunningTime="2026-09-11 00:00:01.140798593 +0000 UTC m=+1404.970625837"
Sep 11 00:00:02.090415 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.090376    2574 generic.go:356] "Generic (PLEG): container finished" podID="9a93e703-9f16-41ae-a5de-89b9723b211a" containerID="cf40454875f8269cf215c696f6fb8503f04c83892ad30d5b77f76dc2f145891f" exitCode=0
Sep 11 00:00:02.090894 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.090469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w" event={"ID":"9a93e703-9f16-41ae-a5de-89b9723b211a","Type":"ContainerDied","Data":"cf40454875f8269cf215c696f6fb8503f04c83892ad30d5b77f76dc2f145891f"}
Sep 11 00:00:02.092193 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.092169    2574 generic.go:356] "Generic (PLEG): container finished" podID="270bda53-cb44-489c-baaa-ca6d6b7ff250" containerID="7dfa26439d4d4bfb95541bb28ee75b7ba61d19ba9758f1e300c339b4dbf42c49" exitCode=0
Sep 11 00:00:02.092303 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.092253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt" event={"ID":"270bda53-cb44-489c-baaa-ca6d6b7ff250","Type":"ContainerDied","Data":"7dfa26439d4d4bfb95541bb28ee75b7ba61d19ba9758f1e300c339b4dbf42c49"}
Sep 11 00:00:02.093915 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.093894    2574 generic.go:356] "Generic (PLEG): container finished" podID="4bdc979f-ab57-49f9-9d20-7b51be5733b4" containerID="614b0734d086da9f27689ddf77b54df1910e388651422f11e2c10bb67e4e42b2" exitCode=0
Sep 11 00:00:02.094032 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.093968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r" event={"ID":"4bdc979f-ab57-49f9-9d20-7b51be5733b4","Type":"ContainerDied","Data":"614b0734d086da9f27689ddf77b54df1910e388651422f11e2c10bb67e4e42b2"}
Sep 11 00:00:02.095463 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.095443    2574 generic.go:356] "Generic (PLEG): container finished" podID="ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb" containerID="d0ee3c26cb8d996e7a6717fd2e4112b148b5f74068ebbe83ab9531b87a7af2ca" exitCode=0
Sep 11 00:00:02.095546 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:02.095497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f" event={"ID":"ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb","Type":"ContainerDied","Data":"d0ee3c26cb8d996e7a6717fd2e4112b148b5f74068ebbe83ab9531b87a7af2ca"}
Sep 11 00:00:03.241699 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.241636    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"
Sep 11 00:00:03.284569 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.284546    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"
Sep 11 00:00:03.313079 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.313053    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"
Sep 11 00:00:03.316541 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.316515    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"
Sep 11 00:00:03.862605 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862571    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"]
Sep 11 00:00:03.862961 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862948    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270bda53-cb44-489c-baaa-ca6d6b7ff250" containerName="cleanup"
Sep 11 00:00:03.862961 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862962    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="270bda53-cb44-489c-baaa-ca6d6b7ff250" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862972    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862978    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862986    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdc979f-ab57-49f9-9d20-7b51be5733b4" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.862992    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdc979f-ab57-49f9-9d20-7b51be5733b4" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863000    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a93e703-9f16-41ae-a5de-89b9723b211a" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863006    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a93e703-9f16-41ae-a5de-89b9723b211a" containerName="cleanup"
Sep 11 00:00:03.863060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863063    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bdc979f-ab57-49f9-9d20-7b51be5733b4" containerName="cleanup"
Sep 11 00:00:03.863302 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863071    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="270bda53-cb44-489c-baaa-ca6d6b7ff250" containerName="cleanup"
Sep 11 00:00:03.863302 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863078    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a93e703-9f16-41ae-a5de-89b9723b211a" containerName="cleanup"
Sep 11 00:00:03.863302 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.863085    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb" containerName="cleanup"
Sep 11 00:00:03.866244 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.866216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:03.868173 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.868154    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:00:03.875130 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.875107    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"]
Sep 11 00:00:03.995014 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:03.994974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8\" (UID: \"599d5e2f-1d9b-4374-b652-222a8c1d6a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:04.096376 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.096338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8\" (UID: \"599d5e2f-1d9b-4374-b652-222a8c1d6a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:04.096973 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.096953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8\" (UID: \"599d5e2f-1d9b-4374-b652-222a8c1d6a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:04.107355 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.107330    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"
Sep 11 00:00:04.107492 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.107329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w" event={"ID":"9a93e703-9f16-41ae-a5de-89b9723b211a","Type":"ContainerDied","Data":"51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc"}
Sep 11 00:00:04.107492 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.107450    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e6977f8dc5519fa32749f995b0058a3326765f214ecc2cfcd2548d313784fc"
Sep 11 00:00:04.108827 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.108799    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"
Sep 11 00:00:04.108827 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.108804    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt" event={"ID":"270bda53-cb44-489c-baaa-ca6d6b7ff250","Type":"ContainerDied","Data":"68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e"}
Sep 11 00:00:04.108987 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.108859    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68cbba6f65104939e9961e3225e3b6307753109d7fc4fc179f70a1513d8fad0e"
Sep 11 00:00:04.110381 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.110356    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"
Sep 11 00:00:04.110381 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.110369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r" event={"ID":"4bdc979f-ab57-49f9-9d20-7b51be5733b4","Type":"ContainerDied","Data":"7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a"}
Sep 11 00:00:04.110522 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.110402    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7f6b43e952e6add2b7bd84283a0716dbb982061d9e2bef1fc5d481b8180e1a"
Sep 11 00:00:04.111835 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.111802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f" event={"ID":"ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb","Type":"ContainerDied","Data":"af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f"}
Sep 11 00:00:04.111947 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.111854    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4a8841b7ddd99be9c7c9cadb21e71031fe84ed0b2ab6f0c3c147150aff9a8f"
Sep 11 00:00:04.111947 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.111813    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"
Sep 11 00:00:04.178328 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.178305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:04.450734 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:04.450709    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"]
Sep 11 00:00:04.453468 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:04.453443    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599d5e2f_1d9b_4374_b652_222a8c1d6a31.slice/crio-f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d WatchSource:0}: Error finding container f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d: Status 404 returned error can't find the container with id f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d
Sep 11 00:00:05.118675 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:05.118635    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8" event={"ID":"599d5e2f-1d9b-4374-b652-222a8c1d6a31","Type":"ContainerStarted","Data":"3b967abd0d3459689692e629d61851bc7f935c4085c1dd062be7127b2b890290"}
Sep 11 00:00:05.122793 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:05.118887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8" event={"ID":"599d5e2f-1d9b-4374-b652-222a8c1d6a31","Type":"ContainerStarted","Data":"f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d"}
Sep 11 00:00:05.143373 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:05.143319    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8" podStartSLOduration=2.143298539 podStartE2EDuration="2.143298539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:05.139221125 +0000 UTC m=+1408.969048346" watchObservedRunningTime="2026-09-11 00:00:05.143298539 +0000 UTC m=+1408.973125761"
Sep 11 00:00:06.126257 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:06.126219    2574 generic.go:356] "Generic (PLEG): container finished" podID="599d5e2f-1d9b-4374-b652-222a8c1d6a31" containerID="3b967abd0d3459689692e629d61851bc7f935c4085c1dd062be7127b2b890290" exitCode=0
Sep 11 00:00:06.126688 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:06.126304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8" event={"ID":"599d5e2f-1d9b-4374-b652-222a8c1d6a31","Type":"ContainerDied","Data":"3b967abd0d3459689692e629d61851bc7f935c4085c1dd062be7127b2b890290"}
Sep 11 00:00:07.268125 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:07.268103    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:07.425102 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:07.425013    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca\") pod \"599d5e2f-1d9b-4374-b652-222a8c1d6a31\" (UID: \"599d5e2f-1d9b-4374-b652-222a8c1d6a31\") "
Sep 11 00:00:07.425470 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:07.425440    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "599d5e2f-1d9b-4374-b652-222a8c1d6a31" (UID: "599d5e2f-1d9b-4374-b652-222a8c1d6a31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:07.525700 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:07.525668    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/599d5e2f-1d9b-4374-b652-222a8c1d6a31-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:08.138645 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:08.138615    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"
Sep 11 00:00:08.138645 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:08.138622    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8" event={"ID":"599d5e2f-1d9b-4374-b652-222a8c1d6a31","Type":"ContainerDied","Data":"f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d"}
Sep 11 00:00:08.138871 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:08.138671    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f08e36d84f4a90cc2f5479c4b969186767ab9bfb734d6d3f1b587798eb33da7d"
Sep 11 00:00:19.312445 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.312411    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"]
Sep 11 00:00:19.314213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.314192    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4ec4c8-29818080-nszgt"]
Sep 11 00:00:19.335706 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.335673    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 11 00:00:19.335960 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.335931    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" podUID="9bcd4b7b-4a77-4929-8114-920fd618a738" containerName="payload-processing" containerID="cri-o://6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7" gracePeriod=30
Sep 11 00:00:19.345627 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.345600    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 11 00:00:19.346476 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.345861    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" podUID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" containerName="payload-pre-processing" containerID="cri-o://9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d" gracePeriod=30
Sep 11 00:00:19.494098 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.494062    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"]
Sep 11 00:00:19.496102 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.496074    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec4c8-dgvc8"]
Sep 11 00:00:19.591998 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.591974    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 11 00:00:19.621984 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.621932    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 11 00:00:19.621984 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.621975    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume\") pod \"9bcd4b7b-4a77-4929-8114-920fd618a738\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") "
Sep 11 00:00:19.622148 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.622102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mfzr\" (UniqueName: \"kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr\") pod \"9bcd4b7b-4a77-4929-8114-920fd618a738\" (UID: \"9bcd4b7b-4a77-4929-8114-920fd618a738\") "
Sep 11 00:00:19.622535 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.622511    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bcd4b7b-4a77-4929-8114-920fd618a738" (UID: "9bcd4b7b-4a77-4929-8114-920fd618a738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:19.624366 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.624338    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr" (OuterVolumeSpecName: "kube-api-access-4mfzr") pod "9bcd4b7b-4a77-4929-8114-920fd618a738" (UID: "9bcd4b7b-4a77-4929-8114-920fd618a738"). InnerVolumeSpecName "kube-api-access-4mfzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:19.723259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.723228    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjmb6\" (UniqueName: \"kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6\") pod \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") "
Sep 11 00:00:19.723424 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.723287    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume\") pod \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\" (UID: \"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9\") "
Sep 11 00:00:19.723656 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.723624    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mfzr\" (UniqueName: \"kubernetes.io/projected/9bcd4b7b-4a77-4929-8114-920fd618a738-kube-api-access-4mfzr\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.723656 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.723651    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bcd4b7b-4a77-4929-8114-920fd618a738-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.723840 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.723677    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" (UID: "bb02eced-7e3f-4b71-a89f-1dc1a90e13c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:19.725397 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.725376    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6" (OuterVolumeSpecName: "kube-api-access-vjmb6") pod "bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" (UID: "bb02eced-7e3f-4b71-a89f-1dc1a90e13c9"). InnerVolumeSpecName "kube-api-access-vjmb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:19.824158 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.824080    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmb6\" (UniqueName: \"kubernetes.io/projected/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-kube-api-access-vjmb6\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:19.824158 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:19.824110    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:20.202785 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.202739    2574 generic.go:356] "Generic (PLEG): container finished" podID="9bcd4b7b-4a77-4929-8114-920fd618a738" containerID="6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7" exitCode=0
Sep 11 00:00:20.203003 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.202814    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"
Sep 11 00:00:20.203003 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.202820    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" event={"ID":"9bcd4b7b-4a77-4929-8114-920fd618a738","Type":"ContainerDied","Data":"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"}
Sep 11 00:00:20.203003 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.202874    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk" event={"ID":"9bcd4b7b-4a77-4929-8114-920fd618a738","Type":"ContainerDied","Data":"4b767a8f78d82aa1ace77ffe75290232b25e3ff552580a3b744354dff1e8c373"}
Sep 11 00:00:20.203003 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.202896    2574 scope.go:117] "RemoveContainer" containerID="6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"
Sep 11 00:00:20.204205 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.204180    2574 generic.go:356] "Generic (PLEG): container finished" podID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" containerID="9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d" exitCode=0
Sep 11 00:00:20.204278 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.204235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" event={"ID":"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9","Type":"ContainerDied","Data":"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"}
Sep 11 00:00:20.204328 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.204290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n" event={"ID":"bb02eced-7e3f-4b71-a89f-1dc1a90e13c9","Type":"ContainerDied","Data":"8e51a29da7bac4f568a6e0d20ff72c5e17bc67c7ae25073b7818b51ff9494109"}
Sep 11 00:00:20.204328 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.204220    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"
Sep 11 00:00:20.213073 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.213057    2574 scope.go:117] "RemoveContainer" containerID="6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"
Sep 11 00:00:20.213398 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:00:20.213371    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7\": container with ID starting with 6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7 not found: ID does not exist" containerID="6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"
Sep 11 00:00:20.213460 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.213407    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7"} err="failed to get container status \"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7\": rpc error: code = NotFound desc = could not find container \"6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7\": container with ID starting with 6551176ce2b21501713c7da5e122f8697559acabf5775281a2cb8d7920c371e7 not found: ID does not exist"
Sep 11 00:00:20.213460 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.213425    2574 scope.go:117] "RemoveContainer" containerID="9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"
Sep 11 00:00:20.222469 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.222451    2574 scope.go:117] "RemoveContainer" containerID="9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"
Sep 11 00:00:20.222725 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:00:20.222706    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d\": container with ID starting with 9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d not found: ID does not exist" containerID="9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"
Sep 11 00:00:20.222790 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.222734    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d"} err="failed to get container status \"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d\": rpc error: code = NotFound desc = could not find container \"9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d\": container with ID starting with 9ed005c73eb202ea9253bf986a59b287c932664ca7b6368a2e2175254d4abb0d not found: ID does not exist"
Sep 11 00:00:20.228477 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.228456    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 11 00:00:20.232436 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.232413    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec4c8-55f75694bb-qpw6n"]
Sep 11 00:00:20.241370 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.241347    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 11 00:00:20.247114 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.247092    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec4c8-7c6855f7ff-sgfgk"]
Sep 11 00:00:20.614210 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.614182    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="270bda53-cb44-489c-baaa-ca6d6b7ff250" path="/var/lib/kubelet/pods/270bda53-cb44-489c-baaa-ca6d6b7ff250/volumes"
Sep 11 00:00:20.614546 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.614494    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="599d5e2f-1d9b-4374-b652-222a8c1d6a31" path="/var/lib/kubelet/pods/599d5e2f-1d9b-4374-b652-222a8c1d6a31/volumes"
Sep 11 00:00:20.614832 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.614819    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcd4b7b-4a77-4929-8114-920fd618a738" path="/var/lib/kubelet/pods/9bcd4b7b-4a77-4929-8114-920fd618a738/volumes"
Sep 11 00:00:20.615139 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:20.615128    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" path="/var/lib/kubelet/pods/bb02eced-7e3f-4b71-a89f-1dc1a90e13c9/volumes"
Sep 11 00:00:26.502171 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502138    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"]
Sep 11 00:00:26.502526 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502482    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" containerName="payload-pre-processing"
Sep 11 00:00:26.502526 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502493    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" containerName="payload-pre-processing"
Sep 11 00:00:26.502526 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502512    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcd4b7b-4a77-4929-8114-920fd618a738" containerName="payload-processing"
Sep 11 00:00:26.502526 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502517    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcd4b7b-4a77-4929-8114-920fd618a738" containerName="payload-processing"
Sep 11 00:00:26.502526 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502524    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599d5e2f-1d9b-4374-b652-222a8c1d6a31" containerName="revoke-keys"
Sep 11 00:00:26.502708 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502530    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="599d5e2f-1d9b-4374-b652-222a8c1d6a31" containerName="revoke-keys"
Sep 11 00:00:26.502708 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502597    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bcd4b7b-4a77-4929-8114-920fd618a738" containerName="payload-processing"
Sep 11 00:00:26.502708 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502606    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="599d5e2f-1d9b-4374-b652-222a8c1d6a31" containerName="revoke-keys"
Sep 11 00:00:26.502708 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.502614    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb02eced-7e3f-4b71-a89f-1dc1a90e13c9" containerName="payload-pre-processing"
Sep 11 00:00:26.507196 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.507171    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:26.510396 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.510363    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:00:26.510524 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.510407    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:00:26.519326 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.519298    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"]
Sep 11 00:00:26.582946 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.582914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw\" (UID: \"ffda738a-a4a6-4b67-93cb-5a6d055f8aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:26.683878 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.683836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw\" (UID: \"ffda738a-a4a6-4b67-93cb-5a6d055f8aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:26.684464 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.684444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw\" (UID: \"ffda738a-a4a6-4b67-93cb-5a6d055f8aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:26.817445 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:26.817362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:27.090284 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:27.090227    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"]
Sep 11 00:00:27.093170 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:27.093146    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffda738a_a4a6_4b67_93cb_5a6d055f8aca.slice/crio-c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39 WatchSource:0}: Error finding container c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39: Status 404 returned error can't find the container with id c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39
Sep 11 00:00:27.246988 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:27.246953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw" event={"ID":"ffda738a-a4a6-4b67-93cb-5a6d055f8aca","Type":"ContainerStarted","Data":"c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39"}
Sep 11 00:00:28.131014 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:28.130977    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 11 00:00:28.253469 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:28.253435    2574 generic.go:356] "Generic (PLEG): container finished" podID="ffda738a-a4a6-4b67-93cb-5a6d055f8aca" containerID="fa8f32d5bf690ea89953632bad6752deaf5d61ecc8fe9c5e43798cc7d1d88eec" exitCode=0
Sep 11 00:00:28.253643 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:28.253521    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw" event={"ID":"ffda738a-a4a6-4b67-93cb-5a6d055f8aca","Type":"ContainerDied","Data":"fa8f32d5bf690ea89953632bad6752deaf5d61ecc8fe9c5e43798cc7d1d88eec"}
Sep 11 00:00:29.006637 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:29.006604    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 11 00:00:29.387430 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:29.387409    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:29.507834 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:29.507803    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca\") pod \"ffda738a-a4a6-4b67-93cb-5a6d055f8aca\" (UID: \"ffda738a-a4a6-4b67-93cb-5a6d055f8aca\") "
Sep 11 00:00:29.508159 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:29.508137    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffda738a-a4a6-4b67-93cb-5a6d055f8aca" (UID: "ffda738a-a4a6-4b67-93cb-5a6d055f8aca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:29.608599 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:29.608567    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffda738a-a4a6-4b67-93cb-5a6d055f8aca-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:30.265147 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:30.265115    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw" event={"ID":"ffda738a-a4a6-4b67-93cb-5a6d055f8aca","Type":"ContainerDied","Data":"c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39"}
Sep 11 00:00:30.265147 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:30.265137    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"
Sep 11 00:00:30.265377 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:30.265170    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15d34b8f0e916ea234b793047e8bdc63feb9396e23bf20c66dbb39702d09e39"
Sep 11 00:00:36.777097 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:36.777064    2574 scope.go:117] "RemoveContainer" containerID="4f3ebea878c05eaae612383de88f9c4b897165e05152e43ef1db97fcb61c54a9"
Sep 11 00:00:36.785315 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:36.785295    2574 scope.go:117] "RemoveContainer" containerID="03f0f946609e11a9d090108dfb2273b261bce0891576ed9e6efbdadbbd0b5fee"
Sep 11 00:00:36.793017 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:36.793000    2574 scope.go:117] "RemoveContainer" containerID="249304f866f525184565bfb41b161ebd85c3134c5487b5acba11b17b4eac537c"
Sep 11 00:00:37.312180 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:37.312146    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"]
Sep 11 00:00:37.313809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:37.313788    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b19d25-29818080-vxn6f"]
Sep 11 00:00:37.506652 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:37.506622    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"]
Sep 11 00:00:37.511455 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:37.511424    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b19d25-jk4dw"]
Sep 11 00:00:38.614051 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:38.614016    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb" path="/var/lib/kubelet/pods/ee755d0c-6fdb-4fd4-bcd5-ff458f1c54cb/volumes"
Sep 11 00:00:38.614426 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:38.614327    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffda738a-a4a6-4b67-93cb-5a6d055f8aca" path="/var/lib/kubelet/pods/ffda738a-a4a6-4b67-93cb-5a6d055f8aca/volumes"
Sep 11 00:00:43.150518 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.150460    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" podUID="0959f228-384a-4314-9f56-b5a3d814a67c" containerName="main" containerID="cri-o://28cb2b379488641986544fb12aee2b4f65a768e2397e6c6a66d963c3666bd0a9" gracePeriod=45
Sep 11 00:00:43.332792 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.332742    2574 generic.go:356] "Generic (PLEG): container finished" podID="0959f228-384a-4314-9f56-b5a3d814a67c" containerID="28cb2b379488641986544fb12aee2b4f65a768e2397e6c6a66d963c3666bd0a9" exitCode=0
Sep 11 00:00:43.332931 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.332815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" event={"ID":"0959f228-384a-4314-9f56-b5a3d814a67c","Type":"ContainerDied","Data":"28cb2b379488641986544fb12aee2b4f65a768e2397e6c6a66d963c3666bd0a9"}
Sep 11 00:00:43.405564 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.405513    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 11 00:00:43.539562 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539534    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.539810 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539570    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.539810 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539594    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.539810 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539657    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.539810 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539680    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.539810 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539703    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgjqp\" (UniqueName: \"kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp\") pod \"0959f228-384a-4314-9f56-b5a3d814a67c\" (UID: \"0959f228-384a-4314-9f56-b5a3d814a67c\") "
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539869    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home" (OuterVolumeSpecName: "home") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539925    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.539946    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.540073    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.540086    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:43.540105 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.540095    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:43.541918 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.541890    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:43.541918 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.541910    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm" (OuterVolumeSpecName: "dshm") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:43.542056 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.541953    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp" (OuterVolumeSpecName: "kube-api-access-cgjqp") pod "0959f228-384a-4314-9f56-b5a3d814a67c" (UID: "0959f228-384a-4314-9f56-b5a3d814a67c"). InnerVolumeSpecName "kube-api-access-cgjqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:43.641120 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.641091    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0959f228-384a-4314-9f56-b5a3d814a67c-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:43.641120 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.641113    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0959f228-384a-4314-9f56-b5a3d814a67c-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:43.641120 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:43.641123    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgjqp\" (UniqueName: \"kubernetes.io/projected/0959f228-384a-4314-9f56-b5a3d814a67c-kube-api-access-cgjqp\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.026241 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.026196    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" podUID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" containerName="main" containerID="cri-o://6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7" gracePeriod=45
Sep 11 00:00:44.278444 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.278386    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 11 00:00:44.338658 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.338629    2574 generic.go:356] "Generic (PLEG): container finished" podID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" containerID="6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7" exitCode=0
Sep 11 00:00:44.338829 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.338670    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"
Sep 11 00:00:44.338829 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.338713    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" event={"ID":"4dc169ab-92c7-4dbf-933e-3eadc64b2adf","Type":"ContainerDied","Data":"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"}
Sep 11 00:00:44.338829 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.338761    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l" event={"ID":"4dc169ab-92c7-4dbf-933e-3eadc64b2adf","Type":"ContainerDied","Data":"682e9d3d1c917d5ccf50ff368179613f2dd9209a72151830603d7af8dba41e3b"}
Sep 11 00:00:44.338829 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.338799    2574 scope.go:117] "RemoveContainer" containerID="6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"
Sep 11 00:00:44.340054 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.340039    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"
Sep 11 00:00:44.340136 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.340083    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc" event={"ID":"0959f228-384a-4314-9f56-b5a3d814a67c","Type":"ContainerDied","Data":"ec775513334764f44b11224c5e68bf37ad61aa6723bbd36804d657d46024f443"}
Sep 11 00:00:44.347750 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.347882 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347759    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.347882 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.347882 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.348128 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347913    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdrm\" (UniqueName: \"kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.348128 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347944    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir\") pod \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\" (UID: \"4dc169ab-92c7-4dbf-933e-3eadc64b2adf\") "
Sep 11 00:00:44.348128 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.347974    2574 scope.go:117] "RemoveContainer" containerID="6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"
Sep 11 00:00:44.348128 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348117    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home" (OuterVolumeSpecName: "home") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:44.348570 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348270    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.348570 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:00:44.348292    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7\": container with ID starting with 6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7 not found: ID does not exist" containerID="6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"
Sep 11 00:00:44.348570 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:44.348570 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348351    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7"} err="failed to get container status \"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7\": rpc error: code = NotFound desc = could not find container \"6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7\": container with ID starting with 6b9ec3c84889f5e1a246a5c1a1232fe35e7cd62f70ebd7a35997468e4d0d2da7 not found: ID does not exist"
Sep 11 00:00:44.348570 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348373    2574 scope.go:117] "RemoveContainer" containerID="28cb2b379488641986544fb12aee2b4f65a768e2397e6c6a66d963c3666bd0a9"
Sep 11 00:00:44.348980 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.348796    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:44.350211 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.350187    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm" (OuterVolumeSpecName: "dshm") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:00:44.350683 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.350665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm" (OuterVolumeSpecName: "kube-api-access-4jdrm") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "kube-api-access-4jdrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:44.350683 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.350671    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4dc169ab-92c7-4dbf-933e-3eadc64b2adf" (UID: "4dc169ab-92c7-4dbf-933e-3eadc64b2adf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:00:44.363169 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.363147    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 11 00:00:44.371774 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.371742    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d7a539/sub-test-model-d7a539-kserve-858d4cd4b5-phrcc"]
Sep 11 00:00:44.384352 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384329    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"]
Sep 11 00:00:44.384708 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384692    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" containerName="main"
Sep 11 00:00:44.384809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384710    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" containerName="main"
Sep 11 00:00:44.384809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384722    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffda738a-a4a6-4b67-93cb-5a6d055f8aca" containerName="revoke-keys"
Sep 11 00:00:44.384809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384730    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffda738a-a4a6-4b67-93cb-5a6d055f8aca" containerName="revoke-keys"
Sep 11 00:00:44.384809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384750    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0959f228-384a-4314-9f56-b5a3d814a67c" containerName="main"
Sep 11 00:00:44.384809 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384759    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0959f228-384a-4314-9f56-b5a3d814a67c" containerName="main"
Sep 11 00:00:44.385063 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384870    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0959f228-384a-4314-9f56-b5a3d814a67c" containerName="main"
Sep 11 00:00:44.385063 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384887    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" containerName="main"
Sep 11 00:00:44.385063 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.384897    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffda738a-a4a6-4b67-93cb-5a6d055f8aca" containerName="revoke-keys"
Sep 11 00:00:44.389292 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.389275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:44.391271 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.391252    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:00:44.391345 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.391320    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:00:44.395584 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.395562    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"]
Sep 11 00:00:44.449528 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.449503    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.449528 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.449526    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.449660 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.449538    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.449660 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.449552    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdrm\" (UniqueName: \"kubernetes.io/projected/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-kube-api-access-4jdrm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.449660 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.449563    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc169ab-92c7-4dbf-933e-3eadc64b2adf-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:44.551020 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.550940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b\" (UID: \"66b142db-772f-474c-bb1f-d7fa50bcf4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:44.615291 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.615249    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0959f228-384a-4314-9f56-b5a3d814a67c" path="/var/lib/kubelet/pods/0959f228-384a-4314-9f56-b5a3d814a67c/volumes"
Sep 11 00:00:44.651829 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.651795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b\" (UID: \"66b142db-772f-474c-bb1f-d7fa50bcf4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:44.652606 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.652582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b\" (UID: \"66b142db-772f-474c-bb1f-d7fa50bcf4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:44.653652 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.653635    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 11 00:00:44.657019 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.656995    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4094ce/sub-test-model-4094ce-kserve-598d77d8cc-zxh2l"]
Sep 11 00:00:44.700263 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.700239    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:44.970589 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:44.970470    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"]
Sep 11 00:00:44.972829 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:00:44.972802    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b142db_772f_474c_bb1f_d7fa50bcf4bc.slice/crio-14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108 WatchSource:0}: Error finding container 14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108: Status 404 returned error can't find the container with id 14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108
Sep 11 00:00:45.349550 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:45.349511    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b" event={"ID":"66b142db-772f-474c-bb1f-d7fa50bcf4bc","Type":"ContainerStarted","Data":"291aa6dd0384d4853edbe75187afaee585c9a8210fc168f66e126bad7ee34842"}
Sep 11 00:00:45.349937 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:45.349612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b" event={"ID":"66b142db-772f-474c-bb1f-d7fa50bcf4bc","Type":"ContainerStarted","Data":"14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108"}
Sep 11 00:00:45.364036 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:45.363985    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b" podStartSLOduration=1.363969469 podStartE2EDuration="1.363969469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:45.362206474 +0000 UTC m=+1449.192033694" watchObservedRunningTime="2026-09-11 00:00:45.363969469 +0000 UTC m=+1449.193796688"
Sep 11 00:00:46.356119 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:46.356083    2574 generic.go:356] "Generic (PLEG): container finished" podID="66b142db-772f-474c-bb1f-d7fa50bcf4bc" containerID="291aa6dd0384d4853edbe75187afaee585c9a8210fc168f66e126bad7ee34842" exitCode=0
Sep 11 00:00:46.356522 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:46.356188    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b" event={"ID":"66b142db-772f-474c-bb1f-d7fa50bcf4bc","Type":"ContainerDied","Data":"291aa6dd0384d4853edbe75187afaee585c9a8210fc168f66e126bad7ee34842"}
Sep 11 00:00:46.615287 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:46.615204    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc169ab-92c7-4dbf-933e-3eadc64b2adf" path="/var/lib/kubelet/pods/4dc169ab-92c7-4dbf-933e-3eadc64b2adf/volumes"
Sep 11 00:00:47.499041 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:47.499018    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:47.580759 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:47.580721    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca\") pod \"66b142db-772f-474c-bb1f-d7fa50bcf4bc\" (UID: \"66b142db-772f-474c-bb1f-d7fa50bcf4bc\") "
Sep 11 00:00:47.581087 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:47.581064    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66b142db-772f-474c-bb1f-d7fa50bcf4bc" (UID: "66b142db-772f-474c-bb1f-d7fa50bcf4bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:47.682331 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:47.682255    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b142db-772f-474c-bb1f-d7fa50bcf4bc-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:48.367916 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:48.367881    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"
Sep 11 00:00:48.367916 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:48.367904    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b" event={"ID":"66b142db-772f-474c-bb1f-d7fa50bcf4bc","Type":"ContainerDied","Data":"14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108"}
Sep 11 00:00:48.368114 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:48.367947    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b88720da43a44b0919500bf0c2cb77b60d28b0623696ef88cc3e8964744108"
Sep 11 00:00:54.829565 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:54.829532    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"]
Sep 11 00:00:54.831823 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:54.831797    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a78403-29818080-9cw4w"]
Sep 11 00:00:54.862878 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:54.862852    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 11 00:00:54.863124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:54.863071    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" podUID="5db21074-d242-47ed-90ed-89e81b166af7" containerName="payload-processing" containerID="cri-o://d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2" gracePeriod=30
Sep 11 00:00:55.104844 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.104816    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"]
Sep 11 00:00:55.110160 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.110138    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a78403-brw8b"]
Sep 11 00:00:55.112223 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.112203    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 11 00:00:55.252092 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.252059    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume\") pod \"5db21074-d242-47ed-90ed-89e81b166af7\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") "
Sep 11 00:00:55.252275 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.252154    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8tw9\" (UniqueName: \"kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9\") pod \"5db21074-d242-47ed-90ed-89e81b166af7\" (UID: \"5db21074-d242-47ed-90ed-89e81b166af7\") "
Sep 11 00:00:55.252467 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.252439    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5db21074-d242-47ed-90ed-89e81b166af7" (UID: "5db21074-d242-47ed-90ed-89e81b166af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:00:55.254244 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.254220    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9" (OuterVolumeSpecName: "kube-api-access-g8tw9") pod "5db21074-d242-47ed-90ed-89e81b166af7" (UID: "5db21074-d242-47ed-90ed-89e81b166af7"). InnerVolumeSpecName "kube-api-access-g8tw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:00:55.352788 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.352681    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db21074-d242-47ed-90ed-89e81b166af7-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:55.352788 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.352711    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tw9\" (UniqueName: \"kubernetes.io/projected/5db21074-d242-47ed-90ed-89e81b166af7-kube-api-access-g8tw9\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:00:55.407285 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.407253    2574 generic.go:356] "Generic (PLEG): container finished" podID="5db21074-d242-47ed-90ed-89e81b166af7" containerID="d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2" exitCode=0
Sep 11 00:00:55.407442 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.407294    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"
Sep 11 00:00:55.407442 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.407345    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" event={"ID":"5db21074-d242-47ed-90ed-89e81b166af7","Type":"ContainerDied","Data":"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"}
Sep 11 00:00:55.407442 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.407405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz" event={"ID":"5db21074-d242-47ed-90ed-89e81b166af7","Type":"ContainerDied","Data":"a0534fa1bca63e074c74fe32e9f806f89eb42a74b54123df62f199cc1faae301"}
Sep 11 00:00:55.407442 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.407434    2574 scope.go:117] "RemoveContainer" containerID="d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"
Sep 11 00:00:55.416547 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.416532    2574 scope.go:117] "RemoveContainer" containerID="d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"
Sep 11 00:00:55.416882 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:00:55.416862    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2\": container with ID starting with d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2 not found: ID does not exist" containerID="d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"
Sep 11 00:00:55.416960 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.416888    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2"} err="failed to get container status \"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2\": rpc error: code = NotFound desc = could not find container \"d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2\": container with ID starting with d94d65b643c0263c1c34d6f6a65fb2f90d549e575eb6e5016e64719fa147dca2 not found: ID does not exist"
Sep 11 00:00:55.430281 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.430252    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 11 00:00:55.434189 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:55.434167    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a78403-54c5f75cd9-vhcfz"]
Sep 11 00:00:56.614629 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:56.614599    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db21074-d242-47ed-90ed-89e81b166af7" path="/var/lib/kubelet/pods/5db21074-d242-47ed-90ed-89e81b166af7/volumes"
Sep 11 00:00:56.615019 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:56.614990    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b142db-772f-474c-bb1f-d7fa50bcf4bc" path="/var/lib/kubelet/pods/66b142db-772f-474c-bb1f-d7fa50bcf4bc/volumes"
Sep 11 00:00:56.615242 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:00:56.615227    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a93e703-9f16-41ae-a5de-89b9723b211a" path="/var/lib/kubelet/pods/9a93e703-9f16-41ae-a5de-89b9723b211a/volumes"
Sep 11 00:01:36.829988 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:01:36.829945    2574 scope.go:117] "RemoveContainer" containerID="8b680f606b3b38105f805918cee58754ee271a2f655b823b207983bf15fe1c2b"
Sep 11 00:03:03.522932 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.522898    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523243    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b142db-772f-474c-bb1f-d7fa50bcf4bc" containerName="revoke-keys"
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523254    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b142db-772f-474c-bb1f-d7fa50bcf4bc" containerName="revoke-keys"
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523263    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db21074-d242-47ed-90ed-89e81b166af7" containerName="payload-processing"
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523270    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db21074-d242-47ed-90ed-89e81b166af7" containerName="payload-processing"
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523317    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db21074-d242-47ed-90ed-89e81b166af7" containerName="payload-processing"
Sep 11 00:03:03.523507 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.523328    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b142db-772f-474c-bb1f-d7fa50bcf4bc" containerName="revoke-keys"
Sep 11 00:03:03.526163 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.526148    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.528905 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.528881    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-17664f\"/\"ipp-route-17664f-kserve-self-signed-certs\""
Sep 11 00:03:03.529036 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.528881    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-17664f\"/\"default-dockercfg-5tf75\""
Sep 11 00:03:03.529036 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.528932    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-17664f\"/\"kube-root-ca.crt\""
Sep 11 00:03:03.529157 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.529058    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-17664f\"/\"openshift-service-ca.crt\""
Sep 11 00:03:03.535356 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.535330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:03:03.542243 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.542370 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.542370 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.542370 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.542370 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.542606 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.542379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss74h\" (UniqueName: \"kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643551 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643702 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss74h\" (UniqueName: \"kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643702 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643702 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643925 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.643925 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.644039 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.643964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.644039 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.644019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.644127 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.644046    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.645945 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.645917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.646194 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.646177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.651319 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.651298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss74h\" (UniqueName: \"kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h\") pod \"ipp-route-17664f-kserve-dd759575f-gv72n\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") " pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:03.835948 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:03.835918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:04.108315 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:04.108292    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:03:04.110660 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:03:04.110634    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d04bc2_08ef_4a66_85fc_a2264c8ee595.slice/crio-52851f36c23bc2881f36a91f4948517d4483a15279b0e7e7a402594b3c64394f WatchSource:0}: Error finding container 52851f36c23bc2881f36a91f4948517d4483a15279b0e7e7a402594b3c64394f: Status 404 returned error can't find the container with id 52851f36c23bc2881f36a91f4948517d4483a15279b0e7e7a402594b3c64394f
Sep 11 00:03:05.059226 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:05.059186    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" event={"ID":"d0d04bc2-08ef-4a66-85fc-a2264c8ee595","Type":"ContainerStarted","Data":"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"}
Sep 11 00:03:05.059592 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:05.059234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" event={"ID":"d0d04bc2-08ef-4a66-85fc-a2264c8ee595","Type":"ContainerStarted","Data":"52851f36c23bc2881f36a91f4948517d4483a15279b0e7e7a402594b3c64394f"}
Sep 11 00:03:05.075571 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:05.075517    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" podStartSLOduration=2.07549865 podStartE2EDuration="2.07549865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:05.074232283 +0000 UTC m=+1588.904059505" watchObservedRunningTime="2026-09-11 00:03:05.07549865 +0000 UTC m=+1588.905325900"
Sep 11 00:03:13.836186 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:13.836144    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:13.836613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:13.836204    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:13.848989 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:13.848967    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:14.116597 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:14.116573    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:03:36.872450 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:36.872379    2574 scope.go:117] "RemoveContainer" containerID="6e59998265cc134712cda7d9993d3ff76aa34fe45786744c29011a9dadc9a09d"
Sep 11 00:03:37.141094 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.141004    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:03:37.146557 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.146531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.148676 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.148654    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ab675d-openshift-default-dockercfg-9xt6z\""
Sep 11 00:03:37.155815 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.155793    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:03:37.236613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236577    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.236831 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pll\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.237173 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.237173 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.236886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338030 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338030 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pll\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338580 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338580 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338691 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338793 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338760    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.338890 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.338868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.339184 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.339161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.340629 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.340610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.340716 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.340668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.346129 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.346109    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.346846 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.346821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pll\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll\") pod \"e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") " pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.458588 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.458509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:37.737013 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.736881    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:03:37.739434 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:03:37.739407    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3656681b_f85c_4797_94a5_43365f3f1ba0.slice/crio-a39cfa01a520ab7b36bd7b55e97e3f278e4ffbfe1447cefcc34d96aff767f587 WatchSource:0}: Error finding container a39cfa01a520ab7b36bd7b55e97e3f278e4ffbfe1447cefcc34d96aff767f587: Status 404 returned error can't find the container with id a39cfa01a520ab7b36bd7b55e97e3f278e4ffbfe1447cefcc34d96aff767f587
Sep 11 00:03:37.742161 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.742121    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:37.742287 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.742215    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:37.742287 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:37.742262    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:38.230469 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:38.230435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerStarted","Data":"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"}
Sep 11 00:03:38.230883 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:38.230488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerStarted","Data":"a39cfa01a520ab7b36bd7b55e97e3f278e4ffbfe1447cefcc34d96aff767f587"}
Sep 11 00:03:38.250493 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:38.250449    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podStartSLOduration=1.250428455 podStartE2EDuration="1.250428455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:38.248962863 +0000 UTC m=+1622.078790083" watchObservedRunningTime="2026-09-11 00:03:38.250428455 +0000 UTC m=+1622.080255675"
Sep 11 00:03:38.458841 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:38.458810    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:38.464273 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:38.464245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:39.235898 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:39.235868    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:39.237355 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:39.237330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:03:42.000468 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.000430    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:03:42.003902 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.003879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.005998 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.005979    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ab675d\""
Sep 11 00:03:42.015248 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.015224    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:03:42.187686 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.187650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxwz\" (UniqueName: \"kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.187686 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.187688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.288525 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.288444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxwz\" (UniqueName: \"kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.288525 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.288481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.289182 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.289164    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.296485 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.296457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxwz\" (UniqueName: \"kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz\") pod \"payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.314309 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.314284    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:42.588837 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:42.588813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:03:42.591195 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:03:42.591166    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e0f9d4_426c_486c_978f_d6748427b382.slice/crio-fb721d7a271e7d220f8c7e6aad01c864fc6bef0a19624a70757c61d288aa79a4 WatchSource:0}: Error finding container fb721d7a271e7d220f8c7e6aad01c864fc6bef0a19624a70757c61d288aa79a4: Status 404 returned error can't find the container with id fb721d7a271e7d220f8c7e6aad01c864fc6bef0a19624a70757c61d288aa79a4
Sep 11 00:03:43.257325 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:43.257283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" event={"ID":"76e0f9d4-426c-486c-978f-d6748427b382","Type":"ContainerStarted","Data":"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"}
Sep 11 00:03:43.257790 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:43.257347    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" event={"ID":"76e0f9d4-426c-486c-978f-d6748427b382","Type":"ContainerStarted","Data":"fb721d7a271e7d220f8c7e6aad01c864fc6bef0a19624a70757c61d288aa79a4"}
Sep 11 00:03:43.257790 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:43.257386    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:43.275048 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:43.274998    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" podStartSLOduration=2.274960247 podStartE2EDuration="2.274960247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:43.274218986 +0000 UTC m=+1627.104046217" watchObservedRunningTime="2026-09-11 00:03:43.274960247 +0000 UTC m=+1627.104787466"
Sep 11 00:03:54.265110 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:54.265080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:03:59.340639 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.340600    2574 generic.go:356] "Generic (PLEG): container finished" podID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerID="6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728" exitCode=0
Sep 11 00:03:59.341111 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.340648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerDied","Data":"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"}
Sep 11 00:03:59.341111 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.341064    2574 scope.go:117] "RemoveContainer" containerID="6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"
Sep 11 00:03:59.342204 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.342175    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:59.342275 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.342263    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:03:59.342314 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:03:59.342296    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 00:04:00.352509 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:00.352469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerStarted","Data":"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"}
Sep 11 00:04:00.458859 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:00.458822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:00.459225 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:00.459194    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:00.459349 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:00.459268    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:01.459561 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:01.459524    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:01.459996 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:01.459589    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:02.459131 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:02.459091    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:02.459327 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:02.459155    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:03.459816 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:03.459748    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:03.460229 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:03.459839    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:04.458928 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:04.458889    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:04.459109 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:04.458947    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:05.459968 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.459928    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 11 00:04:05.460351 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.459990    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 11 00:04:05.796634 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.796557    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"]
Sep 11 00:04:05.799147 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.799131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:05.801276 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.801255    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:04:05.801368 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.801255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:04:05.806727 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.806706    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"]
Sep 11 00:04:05.912306 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:05.912272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr\" (UID: \"37b01d01-4106-4b4a-bf7c-7f28dda162d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:06.013001 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.012968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr\" (UID: \"37b01d01-4106-4b4a-bf7c-7f28dda162d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:06.013540 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.013521    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr\" (UID: \"37b01d01-4106-4b4a-bf7c-7f28dda162d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:06.110348 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.110328    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:06.379129 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.379103    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"]
Sep 11 00:04:06.381973 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:04:06.381943    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b01d01_4106_4b4a_bf7c_7f28dda162d2.slice/crio-288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6 WatchSource:0}: Error finding container 288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6: Status 404 returned error can't find the container with id 288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6
Sep 11 00:04:06.463082 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.463056    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:06.463429 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.463318    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:06.464109 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:06.464094    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:07.390718 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:07.390685    2574 generic.go:356] "Generic (PLEG): container finished" podID="37b01d01-4106-4b4a-bf7c-7f28dda162d2" containerID="a81b477e6004bdf312ee98ee460e2b3da18c74cf6499d1c056feb89110b73f6e" exitCode=0
Sep 11 00:04:07.390854 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:07.390795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr" event={"ID":"37b01d01-4106-4b4a-bf7c-7f28dda162d2","Type":"ContainerDied","Data":"a81b477e6004bdf312ee98ee460e2b3da18c74cf6499d1c056feb89110b73f6e"}
Sep 11 00:04:07.390854 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:07.390843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr" event={"ID":"37b01d01-4106-4b4a-bf7c-7f28dda162d2","Type":"ContainerStarted","Data":"288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6"}
Sep 11 00:04:08.522516 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:08.522488    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:08.636951 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:08.636915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca\") pod \"37b01d01-4106-4b4a-bf7c-7f28dda162d2\" (UID: \"37b01d01-4106-4b4a-bf7c-7f28dda162d2\") "
Sep 11 00:04:08.637247 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:08.637225    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37b01d01-4106-4b4a-bf7c-7f28dda162d2" (UID: "37b01d01-4106-4b4a-bf7c-7f28dda162d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:08.738606 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:08.738527    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b01d01-4106-4b4a-bf7c-7f28dda162d2-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:09.401728 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:09.401697    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"
Sep 11 00:04:09.401910 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:09.401706    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr" event={"ID":"37b01d01-4106-4b4a-bf7c-7f28dda162d2","Type":"ContainerDied","Data":"288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6"}
Sep 11 00:04:09.401910 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:09.401826    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288bb42db85493567999cf691c64cd8ff44761ddf7fd989d557b0ed8469810b6"
Sep 11 00:04:11.969830 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:11.969796    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:04:11.970297 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:11.970045    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" podUID="76e0f9d4-426c-486c-978f-d6748427b382" containerName="payload-pre-processing" containerID="cri-o://d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76" gracePeriod=30
Sep 11 00:04:12.117356 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.117324    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"]
Sep 11 00:04:12.121805 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.121760    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ab675d-pphkr"]
Sep 11 00:04:12.222119 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.222061    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:04:12.372383 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.372354    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rxwz\" (UniqueName: \"kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz\") pod \"76e0f9d4-426c-486c-978f-d6748427b382\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") "
Sep 11 00:04:12.372562 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.372450    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume\") pod \"76e0f9d4-426c-486c-978f-d6748427b382\" (UID: \"76e0f9d4-426c-486c-978f-d6748427b382\") "
Sep 11 00:04:12.372890 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.372866    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76e0f9d4-426c-486c-978f-d6748427b382" (UID: "76e0f9d4-426c-486c-978f-d6748427b382"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:12.374660 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.374639    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz" (OuterVolumeSpecName: "kube-api-access-5rxwz") pod "76e0f9d4-426c-486c-978f-d6748427b382" (UID: "76e0f9d4-426c-486c-978f-d6748427b382"). InnerVolumeSpecName "kube-api-access-5rxwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:12.418852 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.418827    2574 generic.go:356] "Generic (PLEG): container finished" podID="76e0f9d4-426c-486c-978f-d6748427b382" containerID="d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76" exitCode=0
Sep 11 00:04:12.418969 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.418871    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"
Sep 11 00:04:12.418969 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.418896    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" event={"ID":"76e0f9d4-426c-486c-978f-d6748427b382","Type":"ContainerDied","Data":"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"}
Sep 11 00:04:12.418969 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.418937    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9" event={"ID":"76e0f9d4-426c-486c-978f-d6748427b382","Type":"ContainerDied","Data":"fb721d7a271e7d220f8c7e6aad01c864fc6bef0a19624a70757c61d288aa79a4"}
Sep 11 00:04:12.418969 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.418957    2574 scope.go:117] "RemoveContainer" containerID="d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"
Sep 11 00:04:12.428995 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.428980    2574 scope.go:117] "RemoveContainer" containerID="d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"
Sep 11 00:04:12.429282 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:04:12.429256    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76\": container with ID starting with d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76 not found: ID does not exist" containerID="d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"
Sep 11 00:04:12.429401 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.429287    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76"} err="failed to get container status \"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76\": rpc error: code = NotFound desc = could not find container \"d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76\": container with ID starting with d94cfa6e2d96e9d8afb1d3cf093ade1ab4716af7d321765372f8c3305cc23a76 not found: ID does not exist"
Sep 11 00:04:12.438984 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.438961    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:04:12.442625 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.442599    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ab675d-6db9ff5cf4-hbvx9"]
Sep 11 00:04:12.473522 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.473469    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e0f9d4-426c-486c-978f-d6748427b382-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:12.473522 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.473492    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxwz\" (UniqueName: \"kubernetes.io/projected/76e0f9d4-426c-486c-978f-d6748427b382-kube-api-access-5rxwz\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:12.614183 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.614155    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b01d01-4106-4b4a-bf7c-7f28dda162d2" path="/var/lib/kubelet/pods/37b01d01-4106-4b4a-bf7c-7f28dda162d2/volumes"
Sep 11 00:04:12.614516 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:12.614503    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e0f9d4-426c-486c-978f-d6748427b382" path="/var/lib/kubelet/pods/76e0f9d4-426c-486c-978f-d6748427b382/volumes"
Sep 11 00:04:18.437388 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:18.437354    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:04:18.437833 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:18.437637    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" containerID="cri-o://87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d" gracePeriod=30
Sep 11 00:04:21.465910 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:21.465875    2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 00:04:21.466271 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:21.465937    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 00:04:22.123005 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.122970    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"]
Sep 11 00:04:22.123330 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123317    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b01d01-4106-4b4a-bf7c-7f28dda162d2" containerName="revoke-keys"
Sep 11 00:04:22.123374 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123332    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b01d01-4106-4b4a-bf7c-7f28dda162d2" containerName="revoke-keys"
Sep 11 00:04:22.123374 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123347    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e0f9d4-426c-486c-978f-d6748427b382" containerName="payload-pre-processing"
Sep 11 00:04:22.123374 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123353    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e0f9d4-426c-486c-978f-d6748427b382" containerName="payload-pre-processing"
Sep 11 00:04:22.123479 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123414    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e0f9d4-426c-486c-978f-d6748427b382" containerName="payload-pre-processing"
Sep 11 00:04:22.123479 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.123423    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b01d01-4106-4b4a-bf7c-7f28dda162d2" containerName="revoke-keys"
Sep 11 00:04:22.126226 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.126212    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:22.128493 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.128472    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:04:22.128493 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.128486    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:04:22.132609 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.132588    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"]
Sep 11 00:04:22.261272 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.261238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17664f-snw48\" (UID: \"98623777-364e-46c0-9584-8f98d4824042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:22.362613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.362577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17664f-snw48\" (UID: \"98623777-364e-46c0-9584-8f98d4824042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:22.363226 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.363204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17664f-snw48\" (UID: \"98623777-364e-46c0-9584-8f98d4824042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:22.439236 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.439153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:22.707966 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:22.707943    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"]
Sep 11 00:04:22.710433 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:04:22.710403    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98623777_364e_46c0_9584_8f98d4824042.slice/crio-17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de WatchSource:0}: Error finding container 17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de: Status 404 returned error can't find the container with id 17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de
Sep 11 00:04:23.482273 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.482242    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48" event={"ID":"98623777-364e-46c0-9584-8f98d4824042","Type":"ContainerStarted","Data":"3d5f3d6137feed7610d76c95fc947f4cc283f5690b1531c1c668733e44312e97"}
Sep 11 00:04:23.482429 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.482298    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48" event={"ID":"98623777-364e-46c0-9584-8f98d4824042","Type":"ContainerStarted","Data":"17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de"}
Sep 11 00:04:23.688670 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.688644    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:23.711120 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.711024    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48" podStartSLOduration=1.711007357 podStartE2EDuration="1.711007357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:04:23.495673391 +0000 UTC m=+1667.325500611" watchObservedRunningTime="2026-09-11 00:04:23.711007357 +0000 UTC m=+1667.540834577"
Sep 11 00:04:23.877876 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.877841    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2pll\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.877876 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.877881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.877910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.877941    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.877967    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878053    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878095    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878124 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878124    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878443 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878120    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:23.878443 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878147    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs\") pod \"3656681b-f85c-4797-94a5-43365f3f1ba0\" (UID: \"3656681b-f85c-4797-94a5-43365f3f1ba0\") "
Sep 11 00:04:23.878443 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878425    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:23.878613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:23.878613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878486    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.878613 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878588    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data" (OuterVolumeSpecName: "istio-data") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:23.878754 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.878640    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:23.880188 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.880136    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token" (OuterVolumeSpecName: "istio-token") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:23.880188 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.880142    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll" (OuterVolumeSpecName: "kube-api-access-p2pll") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "kube-api-access-p2pll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:23.880463 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.880442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:23.880562 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.880549    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3656681b-f85c-4797-94a5-43365f3f1ba0" (UID: "3656681b-f85c-4797-94a5-43365f3f1ba0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979274    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979297    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979306    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979314    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pll\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-kube-api-access-p2pll\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979323    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979331    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3656681b-f85c-4797-94a5-43365f3f1ba0-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979335 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979339    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:23.979626 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:23.979346    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3656681b-f85c-4797-94a5-43365f3f1ba0-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:24.488679 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.488648    2574 generic.go:356] "Generic (PLEG): container finished" podID="98623777-364e-46c0-9584-8f98d4824042" containerID="3d5f3d6137feed7610d76c95fc947f4cc283f5690b1531c1c668733e44312e97" exitCode=0
Sep 11 00:04:24.488869 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.488736    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48" event={"ID":"98623777-364e-46c0-9584-8f98d4824042","Type":"ContainerDied","Data":"3d5f3d6137feed7610d76c95fc947f4cc283f5690b1531c1c668733e44312e97"}
Sep 11 00:04:24.490495 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.490473    2574 generic.go:356] "Generic (PLEG): container finished" podID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerID="87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d" exitCode=0
Sep 11 00:04:24.490617 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.490517    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"
Sep 11 00:04:24.490617 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.490533    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerDied","Data":"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"}
Sep 11 00:04:24.490617 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.490581    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw" event={"ID":"3656681b-f85c-4797-94a5-43365f3f1ba0","Type":"ContainerDied","Data":"a39cfa01a520ab7b36bd7b55e97e3f278e4ffbfe1447cefcc34d96aff767f587"}
Sep 11 00:04:24.490617 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.490609    2574 scope.go:117] "RemoveContainer" containerID="87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"
Sep 11 00:04:24.500174 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.500157    2574 scope.go:117] "RemoveContainer" containerID="6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"
Sep 11 00:04:24.508919 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.508895    2574 scope.go:117] "RemoveContainer" containerID="87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"
Sep 11 00:04:24.509189 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:04:24.509172    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d\": container with ID starting with 87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d not found: ID does not exist" containerID="87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"
Sep 11 00:04:24.509285 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.509200    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d"} err="failed to get container status \"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d\": rpc error: code = NotFound desc = could not find container \"87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d\": container with ID starting with 87e49db2a0afbeb5f9eca0287d1085d8004c1460ee8355e0005fb8af184bec8d not found: ID does not exist"
Sep 11 00:04:24.509285 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.509223    2574 scope.go:117] "RemoveContainer" containerID="6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"
Sep 11 00:04:24.509497 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:04:24.509472    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728\": container with ID starting with 6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728 not found: ID does not exist" containerID="6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"
Sep 11 00:04:24.509536 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.509509    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728"} err="failed to get container status \"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728\": rpc error: code = NotFound desc = could not find container \"6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728\": container with ID starting with 6c402a8707988a996da8115521c0d793ba069eae87cf7a8bc6444dffd5741728 not found: ID does not exist"
Sep 11 00:04:24.514602 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.514581    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:04:24.517848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.517830    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ab675d-openshift-default-84965994c8-nnbvw"]
Sep 11 00:04:24.614671 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:24.614644    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" path="/var/lib/kubelet/pods/3656681b-f85c-4797-94a5-43365f3f1ba0/volumes"
Sep 11 00:04:25.634117 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:25.634094    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:25.794948 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:25.794858    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca\") pod \"98623777-364e-46c0-9584-8f98d4824042\" (UID: \"98623777-364e-46c0-9584-8f98d4824042\") "
Sep 11 00:04:25.795295 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:25.795266    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98623777-364e-46c0-9584-8f98d4824042" (UID: "98623777-364e-46c0-9584-8f98d4824042"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:04:25.896265 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:25.896236    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98623777-364e-46c0-9584-8f98d4824042-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:26.502838 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:26.502800    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"
Sep 11 00:04:26.502838 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:26.502820    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48" event={"ID":"98623777-364e-46c0-9584-8f98d4824042","Type":"ContainerDied","Data":"17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de"}
Sep 11 00:04:26.503053 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:26.502863    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17dfeac9d1d3972914234cf7a7f34cb53bd1a7b8c5bf943493cc7f147a7f68de"
Sep 11 00:04:37.827789 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:37.827733    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"]
Sep 11 00:04:37.837213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:37.837186    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17664f-snw48"]
Sep 11 00:04:38.613934 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:38.613897    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98623777-364e-46c0-9584-8f98d4824042" path="/var/lib/kubelet/pods/98623777-364e-46c0-9584-8f98d4824042/volumes"
Sep 11 00:04:43.163637 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:43.163603    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:04:58.182576 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.182513    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" podUID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" containerName="main" containerID="cri-o://0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7" gracePeriod=45
Sep 11 00:04:58.430029 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.430002    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:04:58.577137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577052    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577100    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577117    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577365 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss74h\" (UniqueName: \"kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577365 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577260    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577365 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577305    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir\") pod \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\" (UID: \"d0d04bc2-08ef-4a66-85fc-a2264c8ee595\") "
Sep 11 00:04:58.577523 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:58.577523 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577418    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home" (OuterVolumeSpecName: "home") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:58.577636 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577619    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.577686 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577637    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.577686 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.577629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:58.579443 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.579420    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h" (OuterVolumeSpecName: "kube-api-access-ss74h") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "kube-api-access-ss74h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:04:58.579699 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.579679    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm" (OuterVolumeSpecName: "dshm") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:04:58.579776 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.579682    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0d04bc2-08ef-4a66-85fc-a2264c8ee595" (UID: "d0d04bc2-08ef-4a66-85fc-a2264c8ee595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 00:04:58.669287 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.669252    2574 generic.go:356] "Generic (PLEG): container finished" podID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" containerID="0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7" exitCode=0
Sep 11 00:04:58.669454 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.669287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" event={"ID":"d0d04bc2-08ef-4a66-85fc-a2264c8ee595","Type":"ContainerDied","Data":"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"}
Sep 11 00:04:58.669454 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.669318    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"
Sep 11 00:04:58.669454 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.669331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n" event={"ID":"d0d04bc2-08ef-4a66-85fc-a2264c8ee595","Type":"ContainerDied","Data":"52851f36c23bc2881f36a91f4948517d4483a15279b0e7e7a402594b3c64394f"}
Sep 11 00:04:58.669454 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.669350    2574 scope.go:117] "RemoveContainer" containerID="0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"
Sep 11 00:04:58.678140 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678124    2574 scope.go:117] "RemoveContainer" containerID="0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"
Sep 11 00:04:58.678298 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678277    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss74h\" (UniqueName: \"kubernetes.io/projected/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-kube-api-access-ss74h\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.678349 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678304    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.678349 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678316    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.678349 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678324    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d04bc2-08ef-4a66-85fc-a2264c8ee595-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:04:58.678455 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:04:58.678388    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7\": container with ID starting with 0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7 not found: ID does not exist" containerID="0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"
Sep 11 00:04:58.678455 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.678408    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7"} err="failed to get container status \"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7\": rpc error: code = NotFound desc = could not find container \"0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7\": container with ID starting with 0ed803e12cd86600c255c01ae768fc1e8a90fd785435615cf441e3d65ff12af7 not found: ID does not exist"
Sep 11 00:04:58.685353 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.685294    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:04:58.686961 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:04:58.686941    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-17664f/ipp-route-17664f-kserve-dd759575f-gv72n"]
Sep 11 00:05:00.614365 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:00.614285    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" path="/var/lib/kubelet/pods/d0d04bc2-08ef-4a66-85fc-a2264c8ee595/volumes"
Sep 11 00:05:26.404483 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404449    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"]
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404803    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404815    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404826    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" containerName="main"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404831    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" containerName="main"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404845    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98623777-364e-46c0-9584-8f98d4824042" containerName="revoke-keys"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404851    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="98623777-364e-46c0-9584-8f98d4824042" containerName="revoke-keys"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404863    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404869    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404923    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404929    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="98623777-364e-46c0-9584-8f98d4824042" containerName="revoke-keys"
Sep 11 00:05:26.404950 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.404936    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d04bc2-08ef-4a66-85fc-a2264c8ee595" containerName="main"
Sep 11 00:05:26.407868 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.407851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:26.410036 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.410006    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:05:26.410036 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.410030    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:05:26.413795 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.413757    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"]
Sep 11 00:05:26.516115 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.516088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr\" (UID: \"78a09faa-2382-4887-b40e-1883983882b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:26.617203 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.617173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr\" (UID: \"78a09faa-2382-4887-b40e-1883983882b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:26.617736 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.617715    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr\" (UID: \"78a09faa-2382-4887-b40e-1883983882b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:26.719385 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.719310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:26.990377 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:26.990354    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"]
Sep 11 00:05:26.992705 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:05:26.992674    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a09faa_2382_4887_b40e_1883983882b5.slice/crio-370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac WatchSource:0}: Error finding container 370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac: Status 404 returned error can't find the container with id 370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac
Sep 11 00:05:27.822441 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:27.822408    2574 generic.go:356] "Generic (PLEG): container finished" podID="78a09faa-2382-4887-b40e-1883983882b5" containerID="9671ff0077d3e8fea1797b89986cde45f2447fa317cc7d6d8707067fd85c3856" exitCode=0
Sep 11 00:05:27.822837 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:27.822456    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr" event={"ID":"78a09faa-2382-4887-b40e-1883983882b5","Type":"ContainerDied","Data":"9671ff0077d3e8fea1797b89986cde45f2447fa317cc7d6d8707067fd85c3856"}
Sep 11 00:05:27.822837 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:27.822531    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr" event={"ID":"78a09faa-2382-4887-b40e-1883983882b5","Type":"ContainerStarted","Data":"370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac"}
Sep 11 00:05:28.954418 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:28.954387    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:29.039566 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.039532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca\") pod \"78a09faa-2382-4887-b40e-1883983882b5\" (UID: \"78a09faa-2382-4887-b40e-1883983882b5\") "
Sep 11 00:05:29.039953 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.039927    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78a09faa-2382-4887-b40e-1883983882b5" (UID: "78a09faa-2382-4887-b40e-1883983882b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:29.140934 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.140904    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a09faa-2382-4887-b40e-1883983882b5-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:05:29.835373 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.835346    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"
Sep 11 00:05:29.835517 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.835362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr" event={"ID":"78a09faa-2382-4887-b40e-1883983882b5","Type":"ContainerDied","Data":"370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac"}
Sep 11 00:05:29.835517 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:29.835426    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370adadb75e428dfb1882b2039b25f54a07915f9bb6ec268abd67e85ac7772ac"
Sep 11 00:05:37.320539 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:37.320451    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"]
Sep 11 00:05:37.322850 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:37.322820    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f569-ptqpr"]
Sep 11 00:05:38.614456 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:38.614423    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a09faa-2382-4887-b40e-1883983882b5" path="/var/lib/kubelet/pods/78a09faa-2382-4887-b40e-1883983882b5/volumes"
Sep 11 00:05:44.048203 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.048173    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"]
Sep 11 00:05:44.048586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.048511    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a09faa-2382-4887-b40e-1883983882b5" containerName="revoke-keys"
Sep 11 00:05:44.048586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.048521    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a09faa-2382-4887-b40e-1883983882b5" containerName="revoke-keys"
Sep 11 00:05:44.048586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.048582    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3656681b-f85c-4797-94a5-43365f3f1ba0" containerName="istio-proxy"
Sep 11 00:05:44.048690 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.048593    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a09faa-2382-4887-b40e-1883983882b5" containerName="revoke-keys"
Sep 11 00:05:44.051401 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.051386    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:44.053485 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.053458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\""
Sep 11 00:05:44.053485 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.053458    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 00:05:44.059165 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.059127    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"]
Sep 11 00:05:44.176303 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.176264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm\" (UID: \"20cdcbca-e8e4-42e9-9795-e34997992dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:44.276897 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.276865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm\" (UID: \"20cdcbca-e8e4-42e9-9795-e34997992dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:44.277474 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.277455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm\" (UID: \"20cdcbca-e8e4-42e9-9795-e34997992dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:44.363913 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.363890    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:44.631268 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.631239    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"]
Sep 11 00:05:44.633876 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:05:44.633847    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cdcbca_e8e4_42e9_9795_e34997992dae.slice/crio-92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475 WatchSource:0}: Error finding container 92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475: Status 404 returned error can't find the container with id 92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475
Sep 11 00:05:44.915362 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:44.915291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm" event={"ID":"20cdcbca-e8e4-42e9-9795-e34997992dae","Type":"ContainerStarted","Data":"92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475"}
Sep 11 00:05:45.922259 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:45.922224    2574 generic.go:356] "Generic (PLEG): container finished" podID="20cdcbca-e8e4-42e9-9795-e34997992dae" containerID="a3ea301be93deb60692b85b7aa5941518c973fac5c635be3bd14f6d0c81d573d" exitCode=0
Sep 11 00:05:45.922804 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:45.922313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm" event={"ID":"20cdcbca-e8e4-42e9-9795-e34997992dae","Type":"ContainerDied","Data":"a3ea301be93deb60692b85b7aa5941518c973fac5c635be3bd14f6d0c81d573d"}
Sep 11 00:05:47.058760 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.058736    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:47.200906 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.200826    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca\") pod \"20cdcbca-e8e4-42e9-9795-e34997992dae\" (UID: \"20cdcbca-e8e4-42e9-9795-e34997992dae\") "
Sep 11 00:05:47.201173 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.201146    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20cdcbca-e8e4-42e9-9795-e34997992dae" (UID: "20cdcbca-e8e4-42e9-9795-e34997992dae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:47.302225 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.302198    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cdcbca-e8e4-42e9-9795-e34997992dae-maas-api-service-ca\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:05:47.934363 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.934328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"
Sep 11 00:05:47.934537 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.934361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm" event={"ID":"20cdcbca-e8e4-42e9-9795-e34997992dae","Type":"ContainerDied","Data":"92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475"}
Sep 11 00:05:47.934537 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:47.934407    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92260d9e09da95f092b4fb970a594455851e7e45bab310a23487aed8584fb475"
Sep 11 00:05:54.564720 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.564684    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 11 00:05:54.565186 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.564954    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" podUID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" containerName="payload-processing" containerID="cri-o://d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424" gracePeriod=30
Sep 11 00:05:54.718558 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.718528    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"]
Sep 11 00:05:54.722009 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.721983    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d7a539-szghm"]
Sep 11 00:05:54.824504 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.824440    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 11 00:05:54.966692 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.966663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume\") pod \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") "
Sep 11 00:05:54.966883 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.966706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6vsd\" (UniqueName: \"kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd\") pod \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\" (UID: \"1f9d6d8e-941d-44d3-bce2-486b13053fa0\") "
Sep 11 00:05:54.967134 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.967107    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f9d6d8e-941d-44d3-bce2-486b13053fa0" (UID: "1f9d6d8e-941d-44d3-bce2-486b13053fa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:05:54.968960 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.968934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd" (OuterVolumeSpecName: "kube-api-access-c6vsd") pod "1f9d6d8e-941d-44d3-bce2-486b13053fa0" (UID: "1f9d6d8e-941d-44d3-bce2-486b13053fa0"). InnerVolumeSpecName "kube-api-access-c6vsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:05:54.970480 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.970451    2574 generic.go:356] "Generic (PLEG): container finished" podID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" containerID="d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424" exitCode=0
Sep 11 00:05:54.970586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.970499    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"
Sep 11 00:05:54.970586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.970537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" event={"ID":"1f9d6d8e-941d-44d3-bce2-486b13053fa0","Type":"ContainerDied","Data":"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"}
Sep 11 00:05:54.970663 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.970591    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj" event={"ID":"1f9d6d8e-941d-44d3-bce2-486b13053fa0","Type":"ContainerDied","Data":"84214408bb2dd7cf5c3dbaf18f54bb686c245c20761dd37b608380db9434fda4"}
Sep 11 00:05:54.970663 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.970620    2574 scope.go:117] "RemoveContainer" containerID="d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"
Sep 11 00:05:54.980386 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.980368    2574 scope.go:117] "RemoveContainer" containerID="d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"
Sep 11 00:05:54.980651 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:05:54.980628    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424\": container with ID starting with d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424 not found: ID does not exist" containerID="d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"
Sep 11 00:05:54.980804 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.980656    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424"} err="failed to get container status \"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424\": rpc error: code = NotFound desc = could not find container \"d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424\": container with ID starting with d374b6f2d11d3edf7c3b52655b30aba48f6cd60ca940fdc3cdea88a176e51424 not found: ID does not exist"
Sep 11 00:05:54.992015 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.991990    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 11 00:05:54.997945 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:54.997921    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d7a539-765bbbf5b5-ftlwj"]
Sep 11 00:05:55.067992 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:55.067958    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9d6d8e-941d-44d3-bce2-486b13053fa0-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:05:55.067992 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:55.067987    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6vsd\" (UniqueName: \"kubernetes.io/projected/1f9d6d8e-941d-44d3-bce2-486b13053fa0-kube-api-access-c6vsd\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:05:56.615710 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:56.615680    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" path="/var/lib/kubelet/pods/1f9d6d8e-941d-44d3-bce2-486b13053fa0/volumes"
Sep 11 00:05:56.616096 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:05:56.616054    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20cdcbca-e8e4-42e9-9795-e34997992dae" path="/var/lib/kubelet/pods/20cdcbca-e8e4-42e9-9795-e34997992dae/volumes"
Sep 11 00:06:12.261743 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.261703    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"]
Sep 11 00:06:12.264860 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.264833    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4094ce-29818080-7zl6r"]
Sep 11 00:06:12.287573 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.287543    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 11 00:06:12.287910 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.287878    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" podUID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" containerName="payload-processing" containerID="cri-o://3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2" gracePeriod=30
Sep 11 00:06:12.543074 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.543042    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 11 00:06:12.614709 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.614676    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bdc979f-ab57-49f9-9d20-7b51be5733b4" path="/var/lib/kubelet/pods/4bdc979f-ab57-49f9-9d20-7b51be5733b4/volumes"
Sep 11 00:06:12.617533 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.617506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume\") pod \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") "
Sep 11 00:06:12.617682 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.617556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk6jc\" (UniqueName: \"kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc\") pod \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\" (UID: \"4d4cf957-47f0-49bd-8410-d1c9f22e35da\") "
Sep 11 00:06:12.617885 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.617864    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d4cf957-47f0-49bd-8410-d1c9f22e35da" (UID: "4d4cf957-47f0-49bd-8410-d1c9f22e35da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 00:06:12.619648 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.619626    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc" (OuterVolumeSpecName: "kube-api-access-kk6jc") pod "4d4cf957-47f0-49bd-8410-d1c9f22e35da" (UID: "4d4cf957-47f0-49bd-8410-d1c9f22e35da"). InnerVolumeSpecName "kube-api-access-kk6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:06:12.719137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.719110    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4cf957-47f0-49bd-8410-d1c9f22e35da-plugins-config-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:06:12.719137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:12.719135    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6jc\" (UniqueName: \"kubernetes.io/projected/4d4cf957-47f0-49bd-8410-d1c9f22e35da-kube-api-access-kk6jc\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:06:13.058641 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.058601    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" containerID="3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2" exitCode=0
Sep 11 00:06:13.058830 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.058647    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"
Sep 11 00:06:13.058830 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.058662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" event={"ID":"4d4cf957-47f0-49bd-8410-d1c9f22e35da","Type":"ContainerDied","Data":"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"}
Sep 11 00:06:13.058830 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.058712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28" event={"ID":"4d4cf957-47f0-49bd-8410-d1c9f22e35da","Type":"ContainerDied","Data":"d6ecb3809f48a3f71bcd8e6667c69e265604c15cf77dbb06cda119a9408c1d5d"}
Sep 11 00:06:13.058830 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.058742    2574 scope.go:117] "RemoveContainer" containerID="3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"
Sep 11 00:06:13.068449 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.068427    2574 scope.go:117] "RemoveContainer" containerID="3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"
Sep 11 00:06:13.068709 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:06:13.068688    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2\": container with ID starting with 3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2 not found: ID does not exist" containerID="3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"
Sep 11 00:06:13.068797 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.068719    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2"} err="failed to get container status \"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2\": rpc error: code = NotFound desc = could not find container \"3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2\": container with ID starting with 3f7b86109d7880854f08cbb35c4ae1ee8541d8951a4dfa07d7c0e5762e76d8a2 not found: ID does not exist"
Sep 11 00:06:13.080244 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.080214    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 11 00:06:13.084061 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:13.084042    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4094ce-6bd568d679-r6w28"]
Sep 11 00:06:14.614835 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:14.614800    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" path="/var/lib/kubelet/pods/4d4cf957-47f0-49bd-8410-d1c9f22e35da/volumes"
Sep 11 00:06:36.917152 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.917121    2574 scope.go:117] "RemoveContainer" containerID="7dfa26439d4d4bfb95541bb28ee75b7ba61d19ba9758f1e300c339b4dbf42c49"
Sep 11 00:06:36.925166 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.925148    2574 scope.go:117] "RemoveContainer" containerID="d0ee3c26cb8d996e7a6717fd2e4112b148b5f74068ebbe83ab9531b87a7af2ca"
Sep 11 00:06:36.932650 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.932632    2574 scope.go:117] "RemoveContainer" containerID="fafdbc55d9c29cae24d95f07c76ff3c39dc8e49d93875c12bfdaa6b11ccb3d24"
Sep 11 00:06:36.939653 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.939632    2574 scope.go:117] "RemoveContainer" containerID="cf40454875f8269cf215c696f6fb8503f04c83892ad30d5b77f76dc2f145891f"
Sep 11 00:06:36.946666 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.946649    2574 scope.go:117] "RemoveContainer" containerID="fa8f32d5bf690ea89953632bad6752deaf5d61ecc8fe9c5e43798cc7d1d88eec"
Sep 11 00:06:36.953569 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.953555    2574 scope.go:117] "RemoveContainer" containerID="3b967abd0d3459689692e629d61851bc7f935c4085c1dd062be7127b2b890290"
Sep 11 00:06:36.960463 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:06:36.960445    2574 scope.go:117] "RemoveContainer" containerID="614b0734d086da9f27689ddf77b54df1910e388651422f11e2c10bb67e4e42b2"
Sep 11 00:07:37.003572 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:07:37.003541    2574 scope.go:117] "RemoveContainer" containerID="291aa6dd0384d4853edbe75187afaee585c9a8210fc168f66e126bad7ee34842"
Sep 11 00:08:58.503976 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:58.503938    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 11 00:08:58.504540 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:58.504179    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" podUID="fa7aef16-c34c-4a7f-8038-8912c23ad540" containerName="manager" containerID="cri-o://f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e" gracePeriod=10
Sep 11 00:08:59.053806 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.053752    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 11 00:08:59.135087 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.135056    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2622\" (UniqueName: \"kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622\") pod \"fa7aef16-c34c-4a7f-8038-8912c23ad540\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") "
Sep 11 00:08:59.135256 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.135150    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume\") pod \"fa7aef16-c34c-4a7f-8038-8912c23ad540\" (UID: \"fa7aef16-c34c-4a7f-8038-8912c23ad540\") "
Sep 11 00:08:59.135481 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.135457    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fa7aef16-c34c-4a7f-8038-8912c23ad540" (UID: "fa7aef16-c34c-4a7f-8038-8912c23ad540"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 00:08:59.137099 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.137075    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622" (OuterVolumeSpecName: "kube-api-access-n2622") pod "fa7aef16-c34c-4a7f-8038-8912c23ad540" (UID: "fa7aef16-c34c-4a7f-8038-8912c23ad540"). InnerVolumeSpecName "kube-api-access-n2622". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 00:08:59.236846 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.236810    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2622\" (UniqueName: \"kubernetes.io/projected/fa7aef16-c34c-4a7f-8038-8912c23ad540-kube-api-access-n2622\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:08:59.236846 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.236844    2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa7aef16-c34c-4a7f-8038-8912c23ad540-extensions-socket-volume\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\""
Sep 11 00:08:59.819363 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.819323    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa7aef16-c34c-4a7f-8038-8912c23ad540" containerID="f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e" exitCode=0
Sep 11 00:08:59.819848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.819370    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"
Sep 11 00:08:59.819848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.819408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" event={"ID":"fa7aef16-c34c-4a7f-8038-8912c23ad540","Type":"ContainerDied","Data":"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"}
Sep 11 00:08:59.819848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.819460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx" event={"ID":"fa7aef16-c34c-4a7f-8038-8912c23ad540","Type":"ContainerDied","Data":"3b445e932ad82b80a93e21afd55e56b39e5d73870c34debdc797c2468909f0dc"}
Sep 11 00:08:59.819848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.819482    2574 scope.go:117] "RemoveContainer" containerID="f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"
Sep 11 00:08:59.829177 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.829159    2574 scope.go:117] "RemoveContainer" containerID="f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"
Sep 11 00:08:59.829458 ip-10-0-131-31 kubenswrapper[2574]: E0911 00:08:59.829439    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e\": container with ID starting with f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e not found: ID does not exist" containerID="f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"
Sep 11 00:08:59.829529 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.829470    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e"} err="failed to get container status \"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e\": rpc error: code = NotFound desc = could not find container \"f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e\": container with ID starting with f87fc070e7a5d48328e1987a20742da860d27222e880798f972f7f8e6a1cc68e not found: ID does not exist"
Sep 11 00:08:59.840568 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.840538    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 11 00:08:59.844229 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:08:59.844199    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cn8hx"]
Sep 11 00:09:00.614168 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:09:00.614135    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa7aef16-c34c-4a7f-8038-8912c23ad540" path="/var/lib/kubelet/pods/fa7aef16-c34c-4a7f-8038-8912c23ad540/volumes"
Sep 11 00:10:37.036987 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:10:37.036956    2574 scope.go:117] "RemoveContainer" containerID="3d5f3d6137feed7610d76c95fc947f4cc283f5690b1531c1c668733e44312e97"
Sep 11 00:10:37.044664 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:10:37.044641    2574 scope.go:117] "RemoveContainer" containerID="a81b477e6004bdf312ee98ee460e2b3da18c74cf6499d1c056feb89110b73f6e"
Sep 11 00:11:37.067562 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:11:37.067526    2574 scope.go:117] "RemoveContainer" containerID="9671ff0077d3e8fea1797b89986cde45f2447fa317cc7d6d8707067fd85c3856"
Sep 11 00:11:51.930155 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:11:51.930124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bdcb74d9-g99n4_9249f220-6368-4389-98ce-b17ccecfc028/authorino/0.log"
Sep 11 00:11:58.386184 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:11:58.386153    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-wpbs2_23897cad-399e-4809-be7e-2d370ef1e897/copy-manifests/0.log"
Sep 11 00:11:58.398563 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:11:58.398534    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-wpbs2_23897cad-399e-4809-be7e-2d370ef1e897/manager/0.log"
Sep 11 00:11:59.902148 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:11:59.902115    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bdcb74d9-g99n4_9249f220-6368-4389-98ce-b17ccecfc028/authorino/0.log"
Sep 11 00:12:00.249728 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:00.249654    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-slrbk_be189bf2-bdd6-47d2-ba14-c950dfbdf7c2/kuadrant-console-plugin/0.log"
Sep 11 00:12:01.032834 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:01.032802    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p_2d1eccc1-a13d-4628-9d57-85eebaffe30f/istio-proxy/0.log"
Sep 11 00:12:01.391289 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:01.391260    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-f2qxh_1b099978-183b-4d8c-9e00-4b004e5cf76c/kube-auth-proxy/0.log"
Sep 11 00:12:02.340031 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:02.340000    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-c9v7k_4a38bf0d-b624-4da9-aea0-6eef11f0be6a/main/0.log"
Sep 11 00:12:02.563425 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:02.563398    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-llgwd_b6c56541-8428-49a6-b6ea-a3caa02bab04/main/0.log"
Sep 11 00:12:02.783822 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:02.783720    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pw47c_580369c3-9510-4350-a7b2-5fd43897a6c7/main/0.log"
Sep 11 00:12:02.912308 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:02.912281    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-kg8q8_6a4dd1a8-ff15-4edb-8a2b-381b1b8f0437/main/0.log"
Sep 11 00:12:11.492096 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:11.492064    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-528jm_53618be8-c85c-4f61-acb4-1e738be631f0/global-pull-secret-syncer/0.log"
Sep 11 00:12:11.712035 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:11.711992    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mqz5d_1c599127-22a5-4cac-9bb0-48fbc8865853/konnectivity-agent/0.log"
Sep 11 00:12:11.734213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:11.734169    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-31.ec2.internal_6e9761818410ff00337ec1726a7473d4/haproxy/0.log"
Sep 11 00:12:16.354498 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:16.354445    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bdcb74d9-g99n4_9249f220-6368-4389-98ce-b17ccecfc028/authorino/0.log"
Sep 11 00:12:16.428848 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:16.428819    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-slrbk_be189bf2-bdd6-47d2-ba14-c950dfbdf7c2/kuadrant-console-plugin/0.log"
Sep 11 00:12:18.346803 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.346747    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls96s_b1147a04-fea8-4a81-b680-82edc1af9d05/node-exporter/0.log"
Sep 11 00:12:18.373974 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.373937    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls96s_b1147a04-fea8-4a81-b680-82edc1af9d05/kube-rbac-proxy/0.log"
Sep 11 00:12:18.400023 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.400000    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls96s_b1147a04-fea8-4a81-b680-82edc1af9d05/init-textfile/0.log"
Sep 11 00:12:18.669260 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.669228    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/prometheus/0.log"
Sep 11 00:12:18.691142 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.691108    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/config-reloader/0.log"
Sep 11 00:12:18.713490 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.713459    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/thanos-sidecar/0.log"
Sep 11 00:12:18.736457 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.736430    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/kube-rbac-proxy-web/0.log"
Sep 11 00:12:18.760704 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.760678    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/kube-rbac-proxy/0.log"
Sep 11 00:12:18.782505 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.782466    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/kube-rbac-proxy-thanos/0.log"
Sep 11 00:12:18.808242 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:18.808208    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2c38600-cd01-4db0-ad4d-51afa0c2304a/init-config-reloader/0.log"
Sep 11 00:12:20.483285 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:20.483255    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-76jnj_1017b2c6-48eb-4ef0-8a08-642d2402ee55/networking-console-plugin/0.log"
Sep 11 00:12:21.393875 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.393836    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"]
Sep 11 00:12:21.394167 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394154    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7aef16-c34c-4a7f-8038-8912c23ad540" containerName="manager"
Sep 11 00:12:21.394213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394168    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7aef16-c34c-4a7f-8038-8912c23ad540" containerName="manager"
Sep 11 00:12:21.394213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394178    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20cdcbca-e8e4-42e9-9795-e34997992dae" containerName="revoke-keys"
Sep 11 00:12:21.394213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394183    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="20cdcbca-e8e4-42e9-9795-e34997992dae" containerName="revoke-keys"
Sep 11 00:12:21.394213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394194    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" containerName="payload-processing"
Sep 11 00:12:21.394213 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394199    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" containerName="payload-processing"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394214    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" containerName="payload-processing"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394221    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" containerName="payload-processing"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394290    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4cf957-47f0-49bd-8410-d1c9f22e35da" containerName="payload-processing"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394297    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="20cdcbca-e8e4-42e9-9795-e34997992dae" containerName="revoke-keys"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394305    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9d6d8e-941d-44d3-bce2-486b13053fa0" containerName="payload-processing"
Sep 11 00:12:21.394371 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.394312    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7aef16-c34c-4a7f-8038-8912c23ad540" containerName="manager"
Sep 11 00:12:21.397399 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.397377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.399761 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.399735    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"openshift-service-ca.crt\""
Sep 11 00:12:21.400107 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.399750    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rq2f8\"/\"default-dockercfg-7cn6g\""
Sep 11 00:12:21.400252 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.400234    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"kube-root-ca.crt\""
Sep 11 00:12:21.405097 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.405071    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"]
Sep 11 00:12:21.585642 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.585596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-lib-modules\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.586073 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.585657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9jl\" (UniqueName: \"kubernetes.io/projected/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-kube-api-access-5w9jl\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.586073 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.585725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-proc\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.586073 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.585760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-podres\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.586073 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.585857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-sys\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.630176 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.630147    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-zfs8p_57f24d2a-f57a-4a1a-93a5-180bafac42fb/download-server/0.log"
Sep 11 00:12:21.686986 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.686882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9jl\" (UniqueName: \"kubernetes.io/projected/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-kube-api-access-5w9jl\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.686986 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.686928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-proc\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.686986 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.686948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-podres\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.686991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-sys\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.687032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-proc\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.687089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-sys\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.687105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-lib-modules\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.687138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-podres\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.687279 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.687228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-lib-modules\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.694750 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.694721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9jl\" (UniqueName: \"kubernetes.io/projected/9d4b21f5-bd18-472c-af10-86a4f4bc4d75-kube-api-access-5w9jl\") pod \"perf-node-gather-daemonset-qrxw9\" (UID: \"9d4b21f5-bd18-472c-af10-86a4f4bc4d75\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.708360 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.708327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:21.978957 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:21.978899    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"]
Sep 11 00:12:21.981629 ip-10-0-131-31 kubenswrapper[2574]: W0911 00:12:21.981597    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d4b21f5_bd18_472c_af10_86a4f4bc4d75.slice/crio-c38ba94b7c532cc6ce93d7e43934c44b384c4ab7fb06399acdbfd11d55ede527 WatchSource:0}: Error finding container c38ba94b7c532cc6ce93d7e43934c44b384c4ab7fb06399acdbfd11d55ede527: Status 404 returned error can't find the container with id c38ba94b7c532cc6ce93d7e43934c44b384c4ab7fb06399acdbfd11d55ede527
Sep 11 00:12:22.718237 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:22.718197    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9" event={"ID":"9d4b21f5-bd18-472c-af10-86a4f4bc4d75","Type":"ContainerStarted","Data":"cb46edb982107fafba7b2a2affc907debe3ba60a3a462de8af1adc44576b7681"}
Sep 11 00:12:22.718743 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:22.718260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9" event={"ID":"9d4b21f5-bd18-472c-af10-86a4f4bc4d75","Type":"ContainerStarted","Data":"c38ba94b7c532cc6ce93d7e43934c44b384c4ab7fb06399acdbfd11d55ede527"}
Sep 11 00:12:22.718743 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:22.718292    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:22.733575 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:22.733525    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9" podStartSLOduration=1.733507806 podStartE2EDuration="1.733507806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:22.732028197 +0000 UTC m=+2146.561855411" watchObservedRunningTime="2026-09-11 00:12:22.733507806 +0000 UTC m=+2146.563335025"
Sep 11 00:12:23.491397 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:23.491362    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9jhd_de00e982-77f5-4483-84a1-54e3d5c778ba/dns/0.log"
Sep 11 00:12:23.512108 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:23.512078    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9jhd_de00e982-77f5-4483-84a1-54e3d5c778ba/kube-rbac-proxy/0.log"
Sep 11 00:12:23.574060 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:23.574030    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8xph_83a76ea4-06b0-4a7c-9b1f-aa1300d006f1/dns-node-resolver/0.log"
Sep 11 00:12:24.079236 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:24.079203    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64cb7cc9d7-gdr5h_ee3f6c46-0298-4d37-bab0-56acfc42d99a/registry/0.log"
Sep 11 00:12:24.122342 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:24.122294    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4r6t6_6ab405fb-bc66-4ffa-9116-49881c27576a/node-ca/0.log"
Sep 11 00:12:25.025839 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:25.025806    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p_2d1eccc1-a13d-4628-9d57-85eebaffe30f/istio-proxy/0.log"
Sep 11 00:12:25.480271 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:25.480240    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-f2qxh_1b099978-183b-4d8c-9e00-4b004e5cf76c/kube-auth-proxy/0.log"
Sep 11 00:12:26.500857 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:26.500827    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9q52c_9fa81825-3249-49be-8c46-e61dce20883a/serve-healthcheck-canary/0.log"
Sep 11 00:12:27.094643 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:27.094613    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zgzbk_6634f51d-a9bc-43aa-bbe0-dbf3837770ca/kube-rbac-proxy/0.log"
Sep 11 00:12:27.112066 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:27.112038    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zgzbk_6634f51d-a9bc-43aa-bbe0-dbf3837770ca/exporter/0.log"
Sep 11 00:12:27.132094 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:27.132064    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zgzbk_6634f51d-a9bc-43aa-bbe0-dbf3837770ca/extractor/0.log"
Sep 11 00:12:28.733257 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:28.733231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-qrxw9"
Sep 11 00:12:29.900803 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:29.900317    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-wpbs2_23897cad-399e-4809-be7e-2d370ef1e897/manager/0.log"
Sep 11 00:12:29.917591 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:29.917566    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-wpbs2_23897cad-399e-4809-be7e-2d370ef1e897/copy-manifests/0.log"
Sep 11 00:12:31.360114 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:31.360076    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66bf4bb994-4zhhq_7518d434-5693-4795-9901-3c6691ebab73/manager/0.log"
Sep 11 00:12:31.409103 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:31.409061    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ssf8q_d0131d64-a7e2-4ef5-b7f4-fec464243cfa/openshift-lws-operator/0.log"
Sep 11 00:12:31.554916 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:31.554828    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p_2d1eccc1-a13d-4628-9d57-85eebaffe30f/istio-proxy/0.log"
Sep 11 00:12:31.982601 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:31.982560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-f2qxh_1b099978-183b-4d8c-9e00-4b004e5cf76c/kube-auth-proxy/0.log"
Sep 11 00:12:33.617488 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:33.617455    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w9l9p_2d1eccc1-a13d-4628-9d57-85eebaffe30f/istio-proxy/0.log"
Sep 11 00:12:34.061161 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:34.061078    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-f2qxh_1b099978-183b-4d8c-9e00-4b004e5cf76c/kube-auth-proxy/0.log"
Sep 11 00:12:36.273252 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:36.273213    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-c5h8l_855738bf-3722-4a10-9f03-e5f4b9062b1d/migrator/0.log"
Sep 11 00:12:36.291942 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:36.291912    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-c5h8l_855738bf-3722-4a10-9f03-e5f4b9062b1d/graceful-termination/0.log"
Sep 11 00:12:37.088500 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.088461    2574 scope.go:117] "RemoveContainer" containerID="a3ea301be93deb60692b85b7aa5941518c973fac5c635be3bd14f6d0c81d573d"
Sep 11 00:12:37.715261 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.715230    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6rmhq_d567f9a9-22fe-4fc9-82a4-a04294e982e0/kube-multus/0.log"
Sep 11 00:12:37.887099 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.887069    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/kube-multus-additional-cni-plugins/0.log"
Sep 11 00:12:37.907268 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.907238    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/egress-router-binary-copy/0.log"
Sep 11 00:12:37.926622 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.926600    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/cni-plugins/0.log"
Sep 11 00:12:37.946002 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.945973    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/bond-cni-plugin/0.log"
Sep 11 00:12:37.964586 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.964554    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/routeoverride-cni/0.log"
Sep 11 00:12:37.983979 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:37.983900    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/whereabouts-cni-bincopy/0.log"
Sep 11 00:12:38.002926 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:38.002890    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frwgx_5ae77854-f07b-45ff-8684-69dc0ae9d1b2/whereabouts-cni/0.log"
Sep 11 00:12:38.373137 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:38.373107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-msf67_906f16ef-b837-4aac-b339-f93860a0053a/network-metrics-daemon/0.log"
Sep 11 00:12:38.389691 ip-10-0-131-31 kubenswrapper[2574]: I0911 00:12:38.389668    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-msf67_906f16ef-b837-4aac-b339-f93860a0053a/kube-rbac-proxy/0.log"