Sep 16 00:31:50.276761 ip-10-0-134-152 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 00:31:50.276771 ip-10-0-134-152 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 00:31:50.276778 ip-10-0-134-152 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 00:31:50.276987 ip-10-0-134-152 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 00:32:00.282470 ip-10-0-134-152 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 00:32:00.282484 ip-10-0-134-152 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 05fd9a9297384523973f011f8909614b --
Sep 16 00:33:49.709259 ip-10-0-134-152 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 00:33:50.158937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.157613    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 00:33:50.161572 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161553    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.161572 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161571    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.161572 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161575    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.161572 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161578    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161581    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161586    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161590    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161593    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161595    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161598    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161600    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161603    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161606    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161610    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161612    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161615    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161618    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161620    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161623    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161626    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161628    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161631    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161633    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.161691 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161636    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161639    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161641    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161644    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161646    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161648    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161651    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161654    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161657    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161659    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161663    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161665    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161667    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161670    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161673    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161676    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161678    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161681    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161683    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161686    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.162280 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161689    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161692    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161694    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161697    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161701    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161705    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161708    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161711    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161714    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161717    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161720    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161724    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161728    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161730    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161733    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161736    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161738    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161741    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161743    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.162775 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161746    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161749    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161752    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161754    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161756    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161759    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161762    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161766    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161768    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161771    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161773    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161776    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161779    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161781    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161784    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161787    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161789    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161792    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161794    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.163250 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161796    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161799    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161801    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161804    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161806    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161808    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161811    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161813    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161816    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161818    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161821    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161825    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161827    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161830    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161833    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161835    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161838    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.161840    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163365    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163374    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.163716 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163377    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163380    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163384    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163387    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163390    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163393    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163396    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163398    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163401    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163404    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163407    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163409    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163412    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163415    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163418    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163420    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163423    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163426    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163428    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163431    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.164227 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163433    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163436    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163438    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163441    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163444    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163447    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163449    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163452    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163455    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163457    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163461    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163464    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163467    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163471    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163474    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163477    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163479    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163482    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163484    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163487    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.164719 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163489    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163491    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163494    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163497    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163499    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163501    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163504    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163507    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163509    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163511    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163514    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163516    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163519    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163521    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163523    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163526    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163529    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163532    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163535    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.165217 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163539    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163541    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163544    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163546    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163549    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163551    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163554    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163556    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163559    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163562    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163564    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163567    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163569    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163571    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163574    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163576    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163579    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163582    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163584    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.165707 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163588    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163592    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163595    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163598    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163601    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163604    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163607    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163609    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163612    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163615    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163618    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163621    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163623    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163626    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163629    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163631    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163634    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163637    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.163640    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164059    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.166184 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164065    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164068    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164071    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164075    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164079    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164082    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164084    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164087    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164090    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164095    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164098    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164101    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164105    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164108    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164110    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164113    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164116    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164119    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164121    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.166673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164124    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164127    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164130    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164132    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164135    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164137    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164140    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164142    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164145    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164147    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164149    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164152    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164154    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164157    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164159    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164162    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164164    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164167    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164169    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164172    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.167187 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164174    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164177    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164179    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164183    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164185    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164191    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164193    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164196    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164199    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164201    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164204    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164206    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164209    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164211    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164215    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164218    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164221    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164224    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164227    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.167687 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164230    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164233    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164235    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164238    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164241    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164244    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164246    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164248    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164251    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164253    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164256    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164258    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164261    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164263    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164265    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164268    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164270    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164273    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164278    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164280    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.168186 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164283    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164285    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164288    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164290    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164293    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164295    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164298    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164300    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164303    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164305    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164309    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164311    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164314    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164316    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164319    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164321    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164323    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164326    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164328    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.164331    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.168700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164404    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164443    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164449    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164454    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164458    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164461    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164465    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164470    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164473    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164476    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164482    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164485    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164488    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164491    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164494    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164497    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164500    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164503    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164507    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164510    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164513    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164516    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164520    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164523    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 00:33:50.169203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164526    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164529    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164533    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164536    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164539    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164542    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164545    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164548    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164552    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164555    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164558    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164561    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164564    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164567    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164571    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164574    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164578    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164581    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164587    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164591    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164594    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164597    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164600    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164603    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164606    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 00:33:50.169788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164608    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164611    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164614    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164617    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164620    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164623    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164627    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164630    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164633    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164636    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164639    2577 flags.go:64] FLAG: --help="false"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164642    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164645    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164648    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164650    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164654    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164657    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164659    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164662    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164665    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164667    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164670    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164673    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164676    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 00:33:50.170436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164679    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164684    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164687    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164691    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164694    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164697    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164700    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164703    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164709    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164711    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164714    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164717    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164720    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164723    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164726    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164729    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164737    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164740    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164743    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164746    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164749    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164752    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164755    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164758    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164761    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 00:33:50.171078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164763    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164771    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164774    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164777    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164780    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164783    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164788    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164791    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164795    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164799    2577 flags.go:64] FLAG: --port="10250"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164803    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164806    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0210d134d65d77f4a"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164809    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164813    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164816    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164819    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164822    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164825    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164828    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164831    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164834    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164837    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164840    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164843    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164846    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 00:33:50.171700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164849    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164852    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164855    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164857    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164860    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164863    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164866    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164869    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164872    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164874    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164877    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164880    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164883    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164886    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164888    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164895    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164899    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164902    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164907    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164910    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164913    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164916    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164919    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164922    2577 flags.go:64] FLAG: --v="2"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164926    2577 flags.go:64] FLAG: --version="false"
Sep 16 00:33:50.172399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164930    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164934    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.164937    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165067    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165071    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165074    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165077    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165080    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165083    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165085    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165088    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165091    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165094    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165097    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165100    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165103    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165105    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165108    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165111    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.173211 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165113    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165116    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165118    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165122    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165126    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165129    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165132    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165134    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165137    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165140    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165142    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165145    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165149    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165152    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165154    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165157    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165160    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165162    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165164    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165167    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.173936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165169    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165172    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165174    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165177    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165179    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165182    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165184    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165187    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165189    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165192    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165194    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165197    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165199    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165201    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165204    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165209    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165213    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165217    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165220    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.174530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165224    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165227    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165230    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165233    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165235    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165238    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165241    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165243    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165246    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165248    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165250    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165253    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165255    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165258    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165261    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165263    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165265    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165268    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165270    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165273    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.175105 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165277    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165280    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165283    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165285    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165288    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165291    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165294    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165296    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165300    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165304    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165306    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165308    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165311    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165314    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165316    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165320    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165323    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165325    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165327    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165330    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.175661 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165333    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165335    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165338    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.165340    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.166012    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.175516    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.175534    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.175554    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.175561    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175622    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175630    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175633    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175636    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175639    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.176290 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175642    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175645    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175648    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175651    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175653    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175656    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175658    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175661    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175664    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175667    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175670    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175672    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175675    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175678    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175681    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175683    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175687    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175689    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175692    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175694    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.176688 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175697    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175699    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175701    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175704    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175707    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175710    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175712    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175715    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175718    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175733    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175737    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175740    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175743    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175745    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175748    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175751    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175753    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175756    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175759    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.177222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175761    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175764    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175767    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175770    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175773    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175775    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175778    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175781    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175783    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175786    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175788    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175791    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175793    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175796    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175798    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175801    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175803    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175806    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175808    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175811    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.177697 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175814    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175816    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175819    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175821    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175824    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175827    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175830    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175832    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175835    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175837    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175840    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175842    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175845    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175847    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175850    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175853    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175856    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175858    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175861    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175863    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.178209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175865    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175870    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175873    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175876    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175879    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175882    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175885    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175887    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175890    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175893    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175896    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175898    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175901    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175903    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.178703 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.175906    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.175911    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176054    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176060    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176063    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176066    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176069    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176072    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176074    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176077    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176080    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176082    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176085    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176088    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176091    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 00:33:50.179168 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176094    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176096    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176100    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176104    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176107    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176110    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176113    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176116    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176118    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176121    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176124    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176126    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176129    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176132    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176135    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176137    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176141    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176143    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176146    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176148    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 00:33:50.179556 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176151    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176154    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176158    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176161    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176164    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176166    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176168    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176171    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176173    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176176    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176178    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176181    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176184    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176187    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176190    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176192    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176194    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176197    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176199    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176202    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 00:33:50.180086 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176204    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176207    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176209    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176212    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176214    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176217    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176220    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176222    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176225    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176227    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176230    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176232    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176235    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176237    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176240    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176242    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176245    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176247    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176250    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 00:33:50.180614 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176254    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176258    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176261    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176263    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176266    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176269    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176272    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176275    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176277    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176280    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176282    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176285    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176287    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176289    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176292    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176294    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176297    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176300    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176303    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176305    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 00:33:50.181098 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176308    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176310    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176313    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176315    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176318    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176321    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:50.176324    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.176329    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.176452    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.180156    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.181213    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.181319    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:33:50.181588 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.181356    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 00:33:50.184397 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.184381    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:33:50.186629 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.186606    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 00:33:50.200786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.200754    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 00:33:50.204996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.204981    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 00:33:50.205873 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.205846    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 00:33:50.209115 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.209093    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 943c3953-d4fe-47e3-ad7c-ce31b58a61a0:/dev/nvme0n1p4 b6c16c2d-a867-4f02-875f-37fe295d450c:/dev/nvme0n1p3]
Sep 16 00:33:50.209201 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.209113    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 00:33:50.213423 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.213395    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:33:50.215855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.215747    2577 manager.go:216] Machine: {Timestamp:2026-09-16 00:33:50.214402311 +0000 UTC m=+0.375912575 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099245 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fedc96764d0697a6743fa968e9787 SystemUUID:ec2fedc9-6764-d069-7a67-43fa968e9787 BootID:05fd9a92-9738-4523-973f-011f8909614b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:5b:aa:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:5b:aa:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:d4:6a:73:d2:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 00:33:50.215855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.215853    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 00:33:50.215968 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.215940    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 00:33:50.219549 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.219520    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 00:33:50.219686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.219552    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-152.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 00:33:50.220995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.220983    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 00:33:50.221035 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.220997    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 00:33:50.221035 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.221004    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:33:50.221813 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.221803    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 00:33:50.223075 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.223065    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 00:33:50.226702 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226688    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:33:50.226818 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226809    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 00:33:50.226856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226841    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 00:33:50.226856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226852    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 00:33:50.226914 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226862    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 00:33:50.226914 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226872    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 00:33:50.226914 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.226881    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 00:33:50.227341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.227324    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:33:50.227408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.227352    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 00:33:50.229716 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.229696    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 00:33:50.232073 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.232059    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 00:33:50.232129 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.232090    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 00:33:50.234766 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234751    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 00:33:50.234766 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234769    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234775    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234781    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234790    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234796    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234811    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234816    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234823    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234830    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234839    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 00:33:50.234874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.234844    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 00:33:50.236484 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.236461    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 00:33:50.236484 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.236474    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 00:33:50.238935 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.238916    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-152.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 00:33:50.239022 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.238937    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 00:33:50.239022 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.238967    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 00:33:50.240402 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.240390    2577 server.go:1268] "Started kubelet"
Sep 16 00:33:50.240505 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.240480    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 00:33:50.240542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.240497    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 00:33:50.240542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.240524    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 00:33:50.241397 ip-10-0-134-152 systemd[1]: Started Kubernetes Kubelet.
Sep 16 00:33:50.243198 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.243184    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 00:33:50.246112 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.245983    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 00:33:50.246184 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.246129    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 00:33:50.251557 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.251538    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 00:33:50.252187 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.252171    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 00:33:50.252318 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.252282    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.252885 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.252870    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 00:33:50.252985 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.252916    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 00:33:50.252985 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.252928    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 00:33:50.253081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253063    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 00:33:50.253081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253068    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 00:33:50.253259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253240    2577 factory.go:55] Registering systemd factory
Sep 16 00:33:50.253326 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253261    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 00:33:50.253477 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253465    2577 factory.go:153] Registering CRI-O factory
Sep 16 00:33:50.253518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253479    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 00:33:50.253556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253522    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 00:33:50.253556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253540    2577 factory.go:103] Registering Raw factory
Sep 16 00:33:50.253620 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.253558    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 00:33:50.253906 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.252635    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-152.ec2.internal.18d5a5e513596e04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-152.ec2.internal,UID:ip-10-0-134-152.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-152.ec2.internal,},FirstTimestamp:2026-09-16 00:33:50.240361988 +0000 UTC m=+0.401872250,LastTimestamp:2026-09-16 00:33:50.240361988 +0000 UTC m=+0.401872250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-152.ec2.internal,}"
Sep 16 00:33:50.254243 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.254230    2577 manager.go:349] Starting recovery of all containers
Sep 16 00:33:50.258286 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.258260    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5ghx"
Sep 16 00:33:50.265210 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.263471    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 00:33:50.265210 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.263639    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-152.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 00:33:50.265210 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.264441    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 00:33:50.265736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.265646    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5ghx"
Sep 16 00:33:50.267333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.267298    2577 manager.go:354] Recovery completed
Sep 16 00:33:50.271931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.271915    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.273699 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.273684    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.273765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.273715    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.273765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.273729    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.274359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.274344    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 00:33:50.274359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.274355    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 00:33:50.274359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.274359    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.274502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.274370    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 00:33:50.275043 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275028    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.275096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275055    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.275096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275067    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.275669 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275658    2577 policy_none.go:49] "None policy: Start"
Sep 16 00:33:50.275698 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275674    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 00:33:50.275698 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.275685    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 00:33:50.276669 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.276659    2577 policy_none.go:47] "Start"
Sep 16 00:33:50.310629 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.310610    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 00:33:50.310768 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.310652    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 00:33:50.310768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.310668    2577 server.go:87] "Starting device plugin registration server"
Sep 16 00:33:50.311121 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.311062    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 00:33:50.311121 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.311091    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.311291    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.311404    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.311414    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.312579    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.312653    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.326759    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.326805    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.326828    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.326866    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 00:33:50.329315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.328874    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:33:50.411355 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.411243    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.412277 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.412252    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.412388 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.412290    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.412388 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.412303    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.412388 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.412328    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.424707 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.424683    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.424707 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.424712    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-152.ec2.internal\": node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.427809 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.427774    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"]
Sep 16 00:33:50.427887 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.427871    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.429463 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.429446    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.429542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.429477    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.429542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.429488    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.430794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.430771    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.430907 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.430893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.430982 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.430922    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.431648 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431631    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.431750 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431664    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.431750 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431673    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.431750 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431632    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.431750 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431742    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.431882 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.431757    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.432961 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.432936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.433004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.432978    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 00:33:50.433677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.433664    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 00:33:50.433724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.433687    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 00:33:50.433724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.433697    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeHasSufficientPID"
Sep 16 00:33:50.445591 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.445569    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.457697 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.457678    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-152.ec2.internal\" not found" node="ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.462090 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.462072    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-152.ec2.internal\" not found" node="ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.545966 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.545912    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.554567 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.554446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95a97eebf9880fb8b18802bb04714a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-152.ec2.internal\" (UID: \"3d95a97eebf9880fb8b18802bb04714a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.554673 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.554508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.554736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.554694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.646113 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.646068    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.655815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.655977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.655977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95a97eebf9880fb8b18802bb04714a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-152.ec2.internal\" (UID: \"3d95a97eebf9880fb8b18802bb04714a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.655977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95a97eebf9880fb8b18802bb04714a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-152.ec2.internal\" (UID: \"3d95a97eebf9880fb8b18802bb04714a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.655977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.655977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.655906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4faed3c263a4b7e5d401b77a4fbc4407-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal\" (UID: \"4faed3c263a4b7e5d401b77a4fbc4407\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.746251 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.746184    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.760649 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.760615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.766101 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:50.766082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:50.846336 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.846296    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:50.946773 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:50.946737    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:51.047373 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.047295    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:51.147885 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.147852    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-152.ec2.internal\" not found"
Sep 16 00:33:51.181348 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.181314    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 00:33:51.181801 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.181470    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 00:33:51.219561 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.219524    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:33:51.227852 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.227827    2577 apiserver.go:52] "Watching apiserver"
Sep 16 00:33:51.238195 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.238162    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 00:33:51.239358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.239334    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qzgkz","openshift-multus/multus-additional-cni-plugins-5gkht","openshift-multus/multus-n9fp7","openshift-multus/network-metrics-daemon-p9nw7","openshift-network-diagnostics/network-check-target-ftzw9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q","openshift-image-registry/node-ca-ztd8m","openshift-network-operator/iptables-alerter-wxl46","openshift-ovn-kubernetes/ovnkube-node-kvkcq","kube-system/global-pull-secret-syncer-z2xsn","kube-system/konnectivity-agent-xvptl","openshift-cluster-node-tuning-operator/tuned-vrbrp"]
Sep 16 00:33:51.240792 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.240775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.242012 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.241984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.243022 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.242994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.243284 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.243260    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.243384 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.243299    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.243384 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.243271    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndqdx\""
Sep 16 00:33:51.244397 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244331    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 00:33:51.244397 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg2sp\""
Sep 16 00:33:51.244525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.244579 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.244546    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:33:51.244635 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244469    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 00:33:51.244635 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244615    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 00:33:51.244882 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.244865    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.245045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.245031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.245169 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.245146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 00:33:51.245320 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.245290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6wjhh\""
Sep 16 00:33:51.246080 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.246062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:51.246168 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.246147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:33:51.246935 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.246916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.248118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.248103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.249056 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.249036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 00:33:51.249149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.249043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.249395 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.249377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdlmx\""
Sep 16 00:33:51.249478 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.249385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.249698 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.249679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.250554 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.250533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tv9d\""
Sep 16 00:33:51.250677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.250660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 00:33:51.251685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.250843    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.251685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.251024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.251685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.251384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.252129 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.252109    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 00:33:51.252589 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.252572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.252669 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.252604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.252725 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.252703    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:51.253013 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.252991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vl49d\""
Sep 16 00:33:51.253111 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.253040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.253111 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.253059    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 00:33:51.253232 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.253114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:33:51.254018 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.253993    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 00:33:51.254121 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254095    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95m8b\""
Sep 16 00:33:51.254184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 00:33:51.254184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 00:33:51.254281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254243    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.254331 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 00:33:51.254331 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.254307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.255501 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.255480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.256738 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.256723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.257831 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.257813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 00:33:51.257921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.257839    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 00:33:51.257921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.257819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8hlj\""
Sep 16 00:33:51.258539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.258611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-log-socket\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.258611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-script-lib\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.258611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-k8s-cni-cncf-io\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.258611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-multus-certs\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6cd\" (UniqueName: \"kubernetes.io/projected/ad64252b-af91-4f7c-97c7-a9709c37354e-kube-api-access-gr6cd\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6cg\" (UniqueName: \"kubernetes.io/projected/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-kube-api-access-5q6cg\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-device-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-config\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-system-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-socket-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-sys-fs\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad64252b-af91-4f7c-97c7-a9709c37354e-iptables-alerter-script\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b83e392-9d9a-477b-9d05-cd1b5d124150-host\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.258995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.258980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-conf-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b83e392-9d9a-477b-9d05-cd1b5d124150-serviceca\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqggb\" (UniqueName: \"kubernetes.io/projected/3226ed81-5d7e-4d58-837d-99ede28bbcbf-kube-api-access-jqggb\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-cnibin\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-kubelet\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-slash\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-env-overrides\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-etc-kubernetes\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-netns\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.259525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cni-binary-copy\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-socket-dir-parent\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-kubelet\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-hostroot\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-daemon-config\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cnibin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3226ed81-5d7e-4d58-837d-99ede28bbcbf-tmp-dir\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-system-cni-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlm9\" (UniqueName: \"kubernetes.io/projected/803aa2c1-3c35-4b73-98fc-17997ca447c4-kube-api-access-8hlm9\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-var-lib-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-ovn\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsgk\" (UniqueName: \"kubernetes.io/projected/9d763a7a-b3b9-4600-940b-cfcc45815f0b-kube-api-access-vtsgk\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/995499f7-f75d-445e-8ce3-a8b0aa5c860d-agent-certs\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3226ed81-5d7e-4d58-837d-99ede28bbcbf-hosts-file\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-systemd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-node-log\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.260299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmm8\" (UniqueName: \"kubernetes.io/projected/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-kube-api-access-xwmm8\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-etc-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7p7\" (UniqueName: \"kubernetes.io/projected/9b83e392-9d9a-477b-9d05-cd1b5d124150-kube-api-access-bs7p7\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-registration-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad64252b-af91-4f7c-97c7-a9709c37354e-host-slash\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovn-node-metrics-cert\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-kubelet-config\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-dbus\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-netns\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.259984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/995499f7-f75d-445e-8ce3-a8b0aa5c860d-konnectivity-ca\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnmz\" (UniqueName: \"kubernetes.io/projected/0a72feb0-e226-4441-b535-e80f772ad304-kube-api-access-4tnmz\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-netd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-bin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-multus\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.261150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-systemd-units\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.262156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-bin\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.262156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.260124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.262156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.261679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 00:33:51.262156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.261735    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:33:51.262156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.261735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v7zh2\""
Sep 16 00:33:51.264098 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.264070    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 00:33:51.269110 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.269091    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:33:51.269222 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.269170    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"
Sep 16 00:33:51.269358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.269341    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal"]
Sep 16 00:33:51.272264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.272221    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 00:28:50 +0000 UTC" deadline="2028-06-17 18:45:33.194026196 +0000 UTC"
Sep 16 00:33:51.272264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.272259    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15378h11m41.921769403s"
Sep 16 00:33:51.277644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.277625    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 00:33:51.277749 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.277627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal"]
Sep 16 00:33:51.281306 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.281287    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9q99g"
Sep 16 00:33:51.289850 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.289827    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9q99g"
Sep 16 00:33:51.348942 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.348923    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:33:51.353436 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.353414    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 00:33:51.355156 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.355126    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d95a97eebf9880fb8b18802bb04714a.slice/crio-298e5127f8b04e5b90ea71825e333f22502e80c1c95427841cf77ac4d42c0c9c WatchSource:0}: Error finding container 298e5127f8b04e5b90ea71825e333f22502e80c1c95427841cf77ac4d42c0c9c: Status 404 returned error can't find the container with id 298e5127f8b04e5b90ea71825e333f22502e80c1c95427841cf77ac4d42c0c9c
Sep 16 00:33:51.355362 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.355347    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faed3c263a4b7e5d401b77a4fbc4407.slice/crio-51755c8befc8f06a74695393193c74470d11629c604b79dabe5e5e996658a123 WatchSource:0}: Error finding container 51755c8befc8f06a74695393193c74470d11629c604b79dabe5e5e996658a123: Status 404 returned error can't find the container with id 51755c8befc8f06a74695393193c74470d11629c604b79dabe5e5e996658a123
Sep 16 00:33:51.360321 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.360422 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-etc-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.360422 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7p7\" (UniqueName: \"kubernetes.io/projected/9b83e392-9d9a-477b-9d05-cd1b5d124150-kube-api-access-bs7p7\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.360422 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-registration-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.360422 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad64252b-af91-4f7c-97c7-a9709c37354e-host-slash\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovn-node-metrics-cert\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-kubelet-config\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-dbus\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-netns\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-modprobe-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-registration-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.360577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-etc-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-dbus\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-netns\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad64252b-af91-4f7c-97c7-a9709c37354e-host-slash\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-kubernetes\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/995499f7-f75d-445e-8ce3-a8b0aa5c860d-konnectivity-ca\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnmz\" (UniqueName: \"kubernetes.io/projected/0a72feb0-e226-4441-b535-e80f772ad304-kube-api-access-4tnmz\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-run\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-tuned\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6539572a-35ca-4938-a691-21ff110ed047-kubelet-config\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.360931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.360893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.361396 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.361007    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:51.361396 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.361093    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:51.861059967 +0000 UTC m=+2.022570217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:51.361396 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-netd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361396 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361322    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 00:33:51.361396 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-bin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-multus\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-sys\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/995499f7-f75d-445e-8ce3-a8b0aa5c860d-konnectivity-ca\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-systemd-units\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-bin\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-netd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-cni-bin\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-multus\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-cni-bin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-systemd-units\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-log-socket\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-script-lib\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-k8s-cni-cncf-io\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.361864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-multus-certs\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-log-socket\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6cd\" (UniqueName: \"kubernetes.io/projected/ad64252b-af91-4f7c-97c7-a9709c37354e-kube-api-access-gr6cd\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6cg\" (UniqueName: \"kubernetes.io/projected/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-kube-api-access-5q6cg\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-device-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.361986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-config\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-system-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-socket-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-sys-fs\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad64252b-af91-4f7c-97c7-a9709c37354e-iptables-alerter-script\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b83e392-9d9a-477b-9d05-cd1b5d124150-host\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-systemd\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-lib-modules\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-var-lib-kubelet\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-host\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.362256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-conf-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b83e392-9d9a-477b-9d05-cd1b5d124150-serviceca\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-multus-certs\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqggb\" (UniqueName: \"kubernetes.io/projected/3226ed81-5d7e-4d58-837d-99ede28bbcbf-kube-api-access-jqggb\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-cnibin\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-script-lib\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-kubelet\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-slash\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-env-overrides\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-etc-kubernetes\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-conf\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.362874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-netns\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cni-binary-copy\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-device-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-socket-dir-parent\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-conf-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-kubelet\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-hostroot\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-daemon-config\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-tmp\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cnibin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3226ed81-5d7e-4d58-837d-99ede28bbcbf-tmp-dir\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-system-cni-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.363512 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlm9\" (UniqueName: \"kubernetes.io/projected/803aa2c1-3c35-4b73-98fc-17997ca447c4-kube-api-access-8hlm9\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-var-lib-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-ovn\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-cnibin\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsgk\" (UniqueName: \"kubernetes.io/projected/9d763a7a-b3b9-4600-940b-cfcc45815f0b-kube-api-access-vtsgk\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysconfig\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/995499f7-f75d-445e-8ce3-a8b0aa5c860d-agent-certs\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3226ed81-5d7e-4d58-837d-99ede28bbcbf-hosts-file\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ad64252b-af91-4f7c-97c7-a9709c37354e-iptables-alerter-script\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-systemd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-node-log\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.362485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-run-k8s-cni-cncf-io\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a72feb0-e226-4441-b535-e80f772ad304-system-cni-dir\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-var-lib-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b83e392-9d9a-477b-9d05-cd1b5d124150-serviceca\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-ovn\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-etc-kubernetes\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.364279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-hostroot\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-slash\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-host-var-lib-kubelet\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-system-cni-dir\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cnibin\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqpz\" (UniqueName: \"kubernetes.io/projected/341b7d46-4cf5-4a31-89df-6a63e531c14c-kube-api-access-cpqpz\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.363976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmm8\" (UniqueName: \"kubernetes.io/projected/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-kube-api-access-xwmm8\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-systemd\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3226ed81-5d7e-4d58-837d-99ede28bbcbf-tmp-dir\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-daemon-config\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d763a7a-b3b9-4600-940b-cfcc45815f0b-cni-binary-copy\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3226ed81-5d7e-4d58-837d-99ede28bbcbf-hosts-file\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b83e392-9d9a-477b-9d05-cd1b5d124150-host\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-run-netns\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-node-log\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-run-openvswitch\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d763a7a-b3b9-4600-940b-cfcc45815f0b-multus-socket-dir-parent\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-socket-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.364653    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/803aa2c1-3c35-4b73-98fc-17997ca447c4-sys-fs\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.364834    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:51.864816405 +0000 UTC m=+2.026326658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-host-kubelet\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.364969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovnkube-config\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.365288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-env-overrides\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.365870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.365298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a72feb0-e226-4441-b535-e80f772ad304-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.366223 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.366121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-ovn-node-metrics-cert\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.367288 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.367271    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:33:51.368120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.368091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/995499f7-f75d-445e-8ce3-a8b0aa5c860d-agent-certs\") pod \"konnectivity-agent-xvptl\" (UID: \"995499f7-f75d-445e-8ce3-a8b0aa5c860d\") " pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.370851 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.370829    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:33:51.371036 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.370857    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:33:51.371036 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.370873    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:51.371036 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.370938    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:33:51.870918845 +0000 UTC m=+2.032429114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:51.372433 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.372260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7p7\" (UniqueName: \"kubernetes.io/projected/9b83e392-9d9a-477b-9d05-cd1b5d124150-kube-api-access-bs7p7\") pod \"node-ca-ztd8m\" (UID: \"9b83e392-9d9a-477b-9d05-cd1b5d124150\") " pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.372945 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.372917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnmz\" (UniqueName: \"kubernetes.io/projected/0a72feb0-e226-4441-b535-e80f772ad304-kube-api-access-4tnmz\") pod \"multus-additional-cni-plugins-5gkht\" (UID: \"0a72feb0-e226-4441-b535-e80f772ad304\") " pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.373601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.373530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6cg\" (UniqueName: \"kubernetes.io/projected/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-kube-api-access-5q6cg\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.373763 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.373743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6cd\" (UniqueName: \"kubernetes.io/projected/ad64252b-af91-4f7c-97c7-a9709c37354e-kube-api-access-gr6cd\") pod \"iptables-alerter-wxl46\" (UID: \"ad64252b-af91-4f7c-97c7-a9709c37354e\") " pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.374411 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.374391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlm9\" (UniqueName: \"kubernetes.io/projected/803aa2c1-3c35-4b73-98fc-17997ca447c4-kube-api-access-8hlm9\") pod \"aws-ebs-csi-driver-node-znn9q\" (UID: \"803aa2c1-3c35-4b73-98fc-17997ca447c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.374729 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.374708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmm8\" (UniqueName: \"kubernetes.io/projected/2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6-kube-api-access-xwmm8\") pod \"ovnkube-node-kvkcq\" (UID: \"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.374916 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.374898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsgk\" (UniqueName: \"kubernetes.io/projected/9d763a7a-b3b9-4600-940b-cfcc45815f0b-kube-api-access-vtsgk\") pod \"multus-n9fp7\" (UID: \"9d763a7a-b3b9-4600-940b-cfcc45815f0b\") " pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.375918 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.375896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqggb\" (UniqueName: \"kubernetes.io/projected/3226ed81-5d7e-4d58-837d-99ede28bbcbf-kube-api-access-jqggb\") pod \"node-resolver-qzgkz\" (UID: \"3226ed81-5d7e-4d58-837d-99ede28bbcbf\") " pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.383424 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.383405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:33:51.389433 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.389281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:33:51.392791 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.392767    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5040cd_42e7_49ea_bbe3_a13b4af9d8a6.slice/crio-fd23ab4839b0a3fb7360975ec916297dba6b28b8ca8958f75ea05a581c53a162 WatchSource:0}: Error finding container fd23ab4839b0a3fb7360975ec916297dba6b28b8ca8958f75ea05a581c53a162: Status 404 returned error can't find the container with id fd23ab4839b0a3fb7360975ec916297dba6b28b8ca8958f75ea05a581c53a162
Sep 16 00:33:51.397560 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.397537    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995499f7_f75d_445e_8ce3_a8b0aa5c860d.slice/crio-9975b476f68c76b661df4ee0756182d30187f5e38466f3d282ad713a0676bd3c WatchSource:0}: Error finding container 9975b476f68c76b661df4ee0756182d30187f5e38466f3d282ad713a0676bd3c: Status 404 returned error can't find the container with id 9975b476f68c76b661df4ee0756182d30187f5e38466f3d282ad713a0676bd3c
Sep 16 00:33:51.465125 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-modprobe-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465125 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-kubernetes\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-run\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-tuned\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-sys\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-kubernetes\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-sys\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465378 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-run\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-systemd\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-lib-modules\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-var-lib-kubelet\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-host\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-systemd\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-conf\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-host\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-lib-modules\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-modprobe-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-var-lib-kubelet\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-tmp\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.465660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysconfig\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.466152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-conf\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.466152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.466152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysconfig\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.466152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqpz\" (UniqueName: \"kubernetes.io/projected/341b7d46-4cf5-4a31-89df-6a63e531c14c-kube-api-access-cpqpz\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.466152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.465777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-sysctl-d\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.467867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.467852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-etc-tuned\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.467917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.467882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/341b7d46-4cf5-4a31-89df-6a63e531c14c-tmp\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.475048 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.475027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqpz\" (UniqueName: \"kubernetes.io/projected/341b7d46-4cf5-4a31-89df-6a63e531c14c-kube-api-access-cpqpz\") pod \"tuned-vrbrp\" (UID: \"341b7d46-4cf5-4a31-89df-6a63e531c14c\") " pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.576894 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.576810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzgkz"
Sep 16 00:33:51.585427 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.585401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3226ed81_5d7e_4d58_837d_99ede28bbcbf.slice/crio-91ddf03d190e556475bb127ba67e8f0530ae2e6817d085b3a38a04d1b15a8e22 WatchSource:0}: Error finding container 91ddf03d190e556475bb127ba67e8f0530ae2e6817d085b3a38a04d1b15a8e22: Status 404 returned error can't find the container with id 91ddf03d190e556475bb127ba67e8f0530ae2e6817d085b3a38a04d1b15a8e22
Sep 16 00:33:51.592278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.592259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gkht"
Sep 16 00:33:51.596107 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.596085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9fp7"
Sep 16 00:33:51.599519 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.599496    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a72feb0_e226_4441_b535_e80f772ad304.slice/crio-bcf87931ddc27955923f016e7870766ca1bb753ec6cae842aea7a658291578c5 WatchSource:0}: Error finding container bcf87931ddc27955923f016e7870766ca1bb753ec6cae842aea7a658291578c5: Status 404 returned error can't find the container with id bcf87931ddc27955923f016e7870766ca1bb753ec6cae842aea7a658291578c5
Sep 16 00:33:51.603275 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.603244    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d763a7a_b3b9_4600_940b_cfcc45815f0b.slice/crio-c1eca288482e731059fb1d8b4a746d7a67f3ec0b5b13cac28b67dfbc3b0f4f01 WatchSource:0}: Error finding container c1eca288482e731059fb1d8b4a746d7a67f3ec0b5b13cac28b67dfbc3b0f4f01: Status 404 returned error can't find the container with id c1eca288482e731059fb1d8b4a746d7a67f3ec0b5b13cac28b67dfbc3b0f4f01
Sep 16 00:33:51.623286 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.623257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q"
Sep 16 00:33:51.631044 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.631018    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803aa2c1_3c35_4b73_98fc_17997ca447c4.slice/crio-2b264975bdf8466fff0b371a9a44dd564064164bddeaaee41762152d61408817 WatchSource:0}: Error finding container 2b264975bdf8466fff0b371a9a44dd564064164bddeaaee41762152d61408817: Status 404 returned error can't find the container with id 2b264975bdf8466fff0b371a9a44dd564064164bddeaaee41762152d61408817
Sep 16 00:33:51.639861 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.639836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztd8m"
Sep 16 00:33:51.647748 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.647713    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b83e392_9d9a_477b_9d05_cd1b5d124150.slice/crio-9ed647d328fc5e80eb0f4c78cab874a7a9faa9dd82c4c7c9bd91536fca41467e WatchSource:0}: Error finding container 9ed647d328fc5e80eb0f4c78cab874a7a9faa9dd82c4c7c9bd91536fca41467e: Status 404 returned error can't find the container with id 9ed647d328fc5e80eb0f4c78cab874a7a9faa9dd82c4c7c9bd91536fca41467e
Sep 16 00:33:51.668299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.668267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxl46"
Sep 16 00:33:51.676564 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.676527    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad64252b_af91_4f7c_97c7_a9709c37354e.slice/crio-46f77aab044cdfcc12f1ad33b8b59385c64adbb0245a480f31e2359567d64ea3 WatchSource:0}: Error finding container 46f77aab044cdfcc12f1ad33b8b59385c64adbb0245a480f31e2359567d64ea3: Status 404 returned error can't find the container with id 46f77aab044cdfcc12f1ad33b8b59385c64adbb0245a480f31e2359567d64ea3
Sep 16 00:33:51.693585 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.693562    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:33:51.695269 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.695252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vrbrp"
Sep 16 00:33:51.702933 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:33:51.702913    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341b7d46_4cf5_4a31_89df_6a63e531c14c.slice/crio-373416c5070b4853f83807ca93a02e5bb0ce7f5d2c88ddcd3ff3b38c16ec4bfe WatchSource:0}: Error finding container 373416c5070b4853f83807ca93a02e5bb0ce7f5d2c88ddcd3ff3b38c16ec4bfe: Status 404 returned error can't find the container with id 373416c5070b4853f83807ca93a02e5bb0ce7f5d2c88ddcd3ff3b38c16ec4bfe
Sep 16 00:33:51.868040 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.867940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:51.868261 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.868051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:51.868261 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.868085    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:51.868261 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.868151    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:51.868261 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.868159    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:52.868137309 +0000 UTC m=+3.029647583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:51.868261 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.868201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:52.868189244 +0000 UTC m=+3.029699495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:51.968707 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:51.968670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:51.968911 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.968891    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:33:51.969004 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.968917    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:33:51.969004 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.968931    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:51.969096 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:51.969020    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:33:52.968996715 +0000 UTC m=+3.130506976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:52.292359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.292195    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:28:51 +0000 UTC" deadline="2028-03-21 00:05:46.530800891 +0000 UTC"
Sep 16 00:33:52.292359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.292263    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13247h31m54.238542436s"
Sep 16 00:33:52.357394 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.357324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztd8m" event={"ID":"9b83e392-9d9a-477b-9d05-cd1b5d124150","Type":"ContainerStarted","Data":"9ed647d328fc5e80eb0f4c78cab874a7a9faa9dd82c4c7c9bd91536fca41467e"}
Sep 16 00:33:52.357557 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.357418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9fp7" event={"ID":"9d763a7a-b3b9-4600-940b-cfcc45815f0b","Type":"ContainerStarted","Data":"c1eca288482e731059fb1d8b4a746d7a67f3ec0b5b13cac28b67dfbc3b0f4f01"}
Sep 16 00:33:52.367589 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.367546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvptl" event={"ID":"995499f7-f75d-445e-8ce3-a8b0aa5c860d","Type":"ContainerStarted","Data":"9975b476f68c76b661df4ee0756182d30187f5e38466f3d282ad713a0676bd3c"}
Sep 16 00:33:52.374911 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.374871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"fd23ab4839b0a3fb7360975ec916297dba6b28b8ca8958f75ea05a581c53a162"}
Sep 16 00:33:52.392805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.392736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal" event={"ID":"4faed3c263a4b7e5d401b77a4fbc4407","Type":"ContainerStarted","Data":"51755c8befc8f06a74695393193c74470d11629c604b79dabe5e5e996658a123"}
Sep 16 00:33:52.409974 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.409868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vrbrp" event={"ID":"341b7d46-4cf5-4a31-89df-6a63e531c14c","Type":"ContainerStarted","Data":"373416c5070b4853f83807ca93a02e5bb0ce7f5d2c88ddcd3ff3b38c16ec4bfe"}
Sep 16 00:33:52.416814 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.416772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q" event={"ID":"803aa2c1-3c35-4b73-98fc-17997ca447c4","Type":"ContainerStarted","Data":"2b264975bdf8466fff0b371a9a44dd564064164bddeaaee41762152d61408817"}
Sep 16 00:33:52.435139 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.435095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerStarted","Data":"bcf87931ddc27955923f016e7870766ca1bb753ec6cae842aea7a658291578c5"}
Sep 16 00:33:52.441237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.441197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzgkz" event={"ID":"3226ed81-5d7e-4d58-837d-99ede28bbcbf","Type":"ContainerStarted","Data":"91ddf03d190e556475bb127ba67e8f0530ae2e6817d085b3a38a04d1b15a8e22"}
Sep 16 00:33:52.448775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.448738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal" event={"ID":"3d95a97eebf9880fb8b18802bb04714a","Type":"ContainerStarted","Data":"298e5127f8b04e5b90ea71825e333f22502e80c1c95427841cf77ac4d42c0c9c"}
Sep 16 00:33:52.454699 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.454661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxl46" event={"ID":"ad64252b-af91-4f7c-97c7-a9709c37354e","Type":"ContainerStarted","Data":"46f77aab044cdfcc12f1ad33b8b59385c64adbb0245a480f31e2359567d64ea3"}
Sep 16 00:33:52.733832 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.733558    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.877142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.877217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.877364    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.877433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:54.877414726 +0000 UTC m=+5.038924996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.877875    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:52.877964 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.877925    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:54.8779094 +0000 UTC m=+5.039419671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:52.978239 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:52.978161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:52.978445 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.978347    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:33:52.978445 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.978368    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:33:52.978445 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.978381    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:52.978445 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:52.978440    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:33:54.978422337 +0000 UTC m=+5.139932590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:53.293510 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:53.293456    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 00:28:51 +0000 UTC" deadline="2028-03-07 01:17:21.089989249 +0000 UTC"
Sep 16 00:33:53.293510 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:53.293499    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12912h43m27.796495888s"
Sep 16 00:33:53.328209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:53.328173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:53.328411 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:53.328324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:33:53.328832 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:53.328810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:53.328964 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:53.328937    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:33:53.329068 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:53.329013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:53.329126 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:53.329108    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:33:54.894256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:54.894195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:54.894256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:54.894274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:54.894950 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.894433    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:54.894950 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.894502    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:58.894480787 +0000 UTC m=+9.055991053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:54.894950 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.894942    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:54.895145 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.895000    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:33:58.894982957 +0000 UTC m=+9.056493229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:54.995452 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:54.995403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:54.995671 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.995620    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:33:54.995671 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.995646    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:33:54.995671 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.995659    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:54.996105 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:54.995727    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:33:58.995707026 +0000 UTC m=+9.157217296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:55.328171 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:55.328134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:55.328390 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:55.328280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:33:55.328718 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:55.328697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:55.328836 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:55.328810    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:33:55.328915 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:55.328874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:55.329003 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:55.328945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:33:57.328064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:57.328022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:57.328612 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:57.328190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:33:57.328612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:57.328527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:57.328735 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:57.328635    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:33:57.328735 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:57.328681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:57.328849 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:57.328755    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:33:58.929756 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:58.929715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:58.930351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:58.929801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:58.930351 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:58.929879    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:58.930351 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:58.929924    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:58.930351 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:58.929972    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:06.929937966 +0000 UTC m=+17.091448217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:33:58.930351 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:58.929998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:06.929979922 +0000 UTC m=+17.091490189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:33:59.030875 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:59.030837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:59.031032 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.031005    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:33:59.031032 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.031021    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:33:59.031032 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.031030    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:59.031145 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.031077    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:34:07.031062425 +0000 UTC m=+17.192572675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:33:59.327203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:59.327172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:33:59.327381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:59.327257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:33:59.327381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:33:59.327350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:33:59.327381 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.327366    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:33:59.327534 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.327458    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:33:59.327576 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:33:59.327555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:01.327811 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:01.327773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:01.328262 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:01.327901    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:01.328262 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:01.328046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:01.328262 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:01.328143    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:01.328262 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:01.328182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:01.328262 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:01.328248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:03.327907 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:03.327865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:03.328283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:03.327865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:03.328283 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:03.327997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:03.328283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:03.327865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:03.328283 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:03.328053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:03.328283 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:03.328134    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:05.327847 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:05.327814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:05.328328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:05.327815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:05.328328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:05.327815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:05.328328 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:05.327972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:05.328328 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:05.328027    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:05.328328 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:05.328133    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:06.996283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:06.996244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:06.996804 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:06.996315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:06.996804 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:06.996415    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:34:06.996804 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:06.996425    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:34:06.996804 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:06.996482    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:22.996462012 +0000 UTC m=+33.157972871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:34:06.996804 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:06.996502    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:22.996491587 +0000 UTC m=+33.158001845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:34:07.097373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:07.097334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:07.097556 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.097516    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:34:07.097556 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.097547    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:34:07.097657 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.097562    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:34:07.097657 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.097629    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:34:23.09761 +0000 UTC m=+33.259120268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:34:07.327346 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:07.327315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:07.327618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:07.327315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:07.327618 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.327431    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:07.327618 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.327516    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:07.327618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:07.327317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:07.327618 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:07.327611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:09.327206 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:09.327166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:09.327786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:09.327221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:09.327786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:09.327183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:09.327786 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:09.327317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:09.327786 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:09.327393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:09.327786 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:09.327475    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:11.327704 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:11.327661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:11.328095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:11.327677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:11.328095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:11.327782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:11.328095 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:11.327781    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:11.328095 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:11.327854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:11.328095 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:11.327914    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:12.520043 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.519643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9fp7" event={"ID":"9d763a7a-b3b9-4600-940b-cfcc45815f0b","Type":"ContainerStarted","Data":"b6c20e05417854ab06dd2de976801ab7665d98c518665461a89cea3a0905c428"}
Sep 16 00:34:12.521472 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.521441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvptl" event={"ID":"995499f7-f75d-445e-8ce3-a8b0aa5c860d","Type":"ContainerStarted","Data":"e6cb4f40e9ed7f5d6b357b509c70b266ef5d812265872ee5c58afcb11cb0bbc4"}
Sep 16 00:34:12.524673 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"548fff6d8f6a51cec23436f5765dffef78ffe18f988a0e35cae3bf5f3b688c83"}
Sep 16 00:34:12.524791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"5eb81cd7e858b79dc6f3c53949b1e0a2508ff70d5c8e3d23f9cdeaf16a1c2426"}
Sep 16 00:34:12.524791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"51e99f62f2468e08cf9ad8681976165ca725a0e8627f7cabffd53f5f28d2748d"}
Sep 16 00:34:12.524876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"65a2e31275c933bcf92bc63dd38040f6113dc2a105e3052b7a70f51bea18e7fb"}
Sep 16 00:34:12.524918 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"f605ecef03cea2daa5815e0b0016808914d41447c71c5cc22c7e64af7b9a23ab"}
Sep 16 00:34:12.524918 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.524909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"d6aa5d1e701fc25a7dd09c621f6ea187188edc676804d7268717971180125df4"}
Sep 16 00:34:12.526255 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.526229    2577 generic.go:356] "Generic (PLEG): container finished" podID="4faed3c263a4b7e5d401b77a4fbc4407" containerID="5bae5d25d76a070659fd812bcb899bec6f8021777ce8cfc158ab764f2a47e400" exitCode=0
Sep 16 00:34:12.526345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.526298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal" event={"ID":"4faed3c263a4b7e5d401b77a4fbc4407","Type":"ContainerDied","Data":"5bae5d25d76a070659fd812bcb899bec6f8021777ce8cfc158ab764f2a47e400"}
Sep 16 00:34:12.529761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.529730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vrbrp" event={"ID":"341b7d46-4cf5-4a31-89df-6a63e531c14c","Type":"ContainerStarted","Data":"bfe79c0e9162c1fa26a106c592473372c79f6fa4f52841c2d6ab154a913e5529"}
Sep 16 00:34:12.531596 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.531578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q" event={"ID":"803aa2c1-3c35-4b73-98fc-17997ca447c4","Type":"ContainerStarted","Data":"c8bfd57be2db922b731f6462e34c031f12cb6864ab35a6c09bb7c734e7a24f14"}
Sep 16 00:34:12.532997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.532975    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="fd1ce92b497a4c3b77c2c25060cd38d7f48129be7a7e91cc4fdd6ee117d207af" exitCode=0
Sep 16 00:34:12.533077 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.533047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"fd1ce92b497a4c3b77c2c25060cd38d7f48129be7a7e91cc4fdd6ee117d207af"}
Sep 16 00:34:12.534513 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.534483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzgkz" event={"ID":"3226ed81-5d7e-4d58-837d-99ede28bbcbf","Type":"ContainerStarted","Data":"6cb540073b0b399c7215fb6992cc35ace93565a055658a7761b890d59dbbc2f6"}
Sep 16 00:34:12.535885 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.535868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal" event={"ID":"3d95a97eebf9880fb8b18802bb04714a","Type":"ContainerStarted","Data":"d758397255a205ccbdc47bb467246b08c765c630a15bfc8e0c36edffacdbeda9"}
Sep 16 00:34:12.537212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.537189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztd8m" event={"ID":"9b83e392-9d9a-477b-9d05-cd1b5d124150","Type":"ContainerStarted","Data":"443da6d7f180420ca1239276ce1275dd4e0955ac645ab56fc90675f5eb06e4cf"}
Sep 16 00:34:12.538049 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.538012    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n9fp7" podStartSLOduration=1.412263536 podStartE2EDuration="21.537999303s" totalImagesPullingTime="20.125735767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.53766487 +0000 UTC m=+22.699175141" watchObservedRunningTime="2026-09-16 00:34:12.537999303 +0000 UTC m=+22.699509585"
Sep 16 00:34:12.556704 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.556656    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xvptl" podStartSLOduration=1.351080731 podStartE2EDuration="21.556641619s" totalImagesPullingTime="20.205560888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.556179692 +0000 UTC m=+22.717689964" watchObservedRunningTime="2026-09-16 00:34:12.556641619 +0000 UTC m=+22.718151890"
Sep 16 00:34:12.585226 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.585169    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vrbrp" podStartSLOduration=1.683401666 podStartE2EDuration="21.585147167s" totalImagesPullingTime="19.901745501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.571162211 +0000 UTC m=+22.732672483" watchObservedRunningTime="2026-09-16 00:34:12.585147167 +0000 UTC m=+22.746657431"
Sep 16 00:34:12.625167 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.625125    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-152.ec2.internal" podStartSLOduration=21.625109432 podStartE2EDuration="21.625109432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.606800081 +0000 UTC m=+22.768310355" watchObservedRunningTime="2026-09-16 00:34:12.625109432 +0000 UTC m=+22.786619723"
Sep 16 00:34:12.638111 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.638066    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qzgkz" podStartSLOduration=2.619833613 podStartE2EDuration="22.63805005s" totalImagesPullingTime="20.018216437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.624517262 +0000 UTC m=+22.786027553" watchObservedRunningTime="2026-09-16 00:34:12.63805005 +0000 UTC m=+22.799560337"
Sep 16 00:34:12.638427 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:12.638404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztd8m" podStartSLOduration=1.683299131 podStartE2EDuration="21.63839794s" totalImagesPullingTime="19.955098809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:12.638033107 +0000 UTC m=+22.799543379" watchObservedRunningTime="2026-09-16 00:34:12.63839794 +0000 UTC m=+22.799908212"
Sep 16 00:34:13.327568 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.327532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:13.327759 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.327540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:13.327759 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:13.327647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:13.327759 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.327671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:13.327929 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:13.327784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:13.327929 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:13.327874    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:13.544123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.544085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal" event={"ID":"4faed3c263a4b7e5d401b77a4fbc4407","Type":"ContainerStarted","Data":"98a826d2cd6927cc63526e008e193bcf593d1f39209eee46c467ce10b5ec7971"}
Sep 16 00:34:13.548151 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.547737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxl46" event={"ID":"ad64252b-af91-4f7c-97c7-a9709c37354e","Type":"ContainerStarted","Data":"1070af33639e5a4cce53d55451b0604c78237238ec757daa1599c89ce05b9539"}
Sep 16 00:34:13.558690 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.558635    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-152.ec2.internal" podStartSLOduration=22.558614486 podStartE2EDuration="22.558614486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:13.557864883 +0000 UTC m=+23.719375154" watchObservedRunningTime="2026-09-16 00:34:13.558614486 +0000 UTC m=+23.720124758"
Sep 16 00:34:13.678317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.678286    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 00:34:13.781216 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.781177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:34:13.781938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.781915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:34:13.797908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:13.797854    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxl46" podStartSLOduration=2.87107414 podStartE2EDuration="22.797838838s" totalImagesPullingTime="19.926764698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:13.571516067 +0000 UTC m=+23.733026342" watchObservedRunningTime="2026-09-16 00:34:13.797838838 +0000 UTC m=+23.959349113"
Sep 16 00:34:14.324235 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.324102    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T00:34:13.678310258Z","UUID":"f3f2e79b-5f08-4cac-9a59-9e73f48e798a","Handler":null,"Name":"","Endpoint":""}
Sep 16 00:34:14.326321 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.326297    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 00:34:14.326450 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.326337    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 00:34:14.554115 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.553882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q" event={"ID":"803aa2c1-3c35-4b73-98fc-17997ca447c4","Type":"ContainerStarted","Data":"78ecdf2bf71b7fde18d1e6d71438c716cfd3f0053bb52bab6285bf733cb7de1a"}
Sep 16 00:34:14.554848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.554826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:34:14.554975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:14.554911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvptl"
Sep 16 00:34:15.328009 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.327971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:15.328009 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.328010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:15.328276 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.328084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:15.328276 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:15.328106    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:15.328276 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:15.328165    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:15.328276 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:15.328240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:15.561026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.560979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"ffa8bf2672050886d85c703e9a4b378c50a8e795418f73fa82d298df233adabe"}
Sep 16 00:34:15.563391 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.563356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q" event={"ID":"803aa2c1-3c35-4b73-98fc-17997ca447c4","Type":"ContainerStarted","Data":"0f48bfc7fa299c71e9d0b70b5f36dca07d0979aa0ffa7bf44a6884286b8dab47"}
Sep 16 00:34:15.580844 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:15.580746    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znn9q" podStartSLOduration=2.183415796 podStartE2EDuration="24.580724242s" totalImagesPullingTime="22.397308446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:15.579481847 +0000 UTC m=+25.740992120" watchObservedRunningTime="2026-09-16 00:34:15.580724242 +0000 UTC m=+25.742234515"
Sep 16 00:34:17.327414 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:17.327377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:17.327414 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:17.327416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:17.327902 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:17.327377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:17.327902 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:17.327500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:17.327902 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:17.327627    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:17.327902 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:17.327724    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:18.572575 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.572399    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="9c9675cfeea37cf1e506822e88afd5b6bf3dda7fba2bcdca85a8fd1cd5d8bd0d" exitCode=0
Sep 16 00:34:18.573026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.572475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"9c9675cfeea37cf1e506822e88afd5b6bf3dda7fba2bcdca85a8fd1cd5d8bd0d"}
Sep 16 00:34:18.576432 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.576405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" event={"ID":"2a5040cd-42e7-49ea-bbe3-a13b4af9d8a6","Type":"ContainerStarted","Data":"8ff5af7843fc97cfabf3921f6aecf6dec8ffd47681b93d4da42e7bfd5c8bf550"}
Sep 16 00:34:18.576676 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.576661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:18.576747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.576708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:18.592621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.592587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:18.613335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:18.613290    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq" podStartSLOduration=7.370101612 podStartE2EDuration="27.613275462s" totalImagesPullingTime="20.24317385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:18.612660126 +0000 UTC m=+28.774170397" watchObservedRunningTime="2026-09-16 00:34:18.613275462 +0000 UTC m=+28.774785747"
Sep 16 00:34:19.327170 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.327138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:19.327334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.327146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:19.327334 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.327287    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:19.327334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.327146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:19.327454 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.327354    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:19.327454 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.327406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:19.581435 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.581346    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="3f4bc9850a558b830def435560cfafceb33fa1367786714fb2a0d8ed1bfe5ccd" exitCode=0
Sep 16 00:34:19.581435 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.581426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"3f4bc9850a558b830def435560cfafceb33fa1367786714fb2a0d8ed1bfe5ccd"}
Sep 16 00:34:19.582186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.581885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:19.598640 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.598614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:19.787405 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.787369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2xsn"]
Sep 16 00:34:19.787591 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.787515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:19.787663 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.787608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:19.790913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.790879    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ftzw9"]
Sep 16 00:34:19.791123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.791027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:19.791197 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.791164    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:19.791574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.791552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9nw7"]
Sep 16 00:34:19.791683 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:19.791648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:19.791764 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:19.791744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:20.586502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:20.586471    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="807a3de5f5a1b1047f9d6f01d3e29a6bdf94c3d915033128effb2791e80ce192" exitCode=0
Sep 16 00:34:20.587100 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:20.586560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"807a3de5f5a1b1047f9d6f01d3e29a6bdf94c3d915033128effb2791e80ce192"}
Sep 16 00:34:21.327302 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:21.327264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:21.327448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:21.327273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:21.327448 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:21.327372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:21.327538 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:21.327456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:21.327538 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:21.327273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:21.327657 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:21.327601    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:23.008619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.008411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:23.009023 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.008654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:23.009023 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.008568    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:34:23.009023 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.008740    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs podName:1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:55.008724667 +0000 UTC m=+65.170234917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs") pod "network-metrics-daemon-p9nw7" (UID: "1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 00:34:23.009023 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.008753    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 00:34:23.009023 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.008806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret podName:6539572a-35ca-4938-a691-21ff110ed047 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:55.008791976 +0000 UTC m=+65.170302226 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret") pod "global-pull-secret-syncer-z2xsn" (UID: "6539572a-35ca-4938-a691-21ff110ed047") : object "kube-system"/"original-pull-secret" not registered
Sep 16 00:34:23.109858 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.109817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:23.110051 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.109999    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 00:34:23.110051 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.110015    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 00:34:23.110051 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.110026    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gfgzt for pod openshift-network-diagnostics/network-check-target-ftzw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:34:23.110228 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.110084    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt podName:1f59ef6a-348f-4b5b-9831-e8f3038a4d1e nodeName:}" failed. No retries permitted until 2026-09-16 00:34:55.110064027 +0000 UTC m=+65.271574293 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfgzt" (UniqueName: "kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt") pod "network-check-target-ftzw9" (UID: "1f59ef6a-348f-4b5b-9831-e8f3038a4d1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 00:34:23.327910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.327831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:23.328075 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.327831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:23.328075 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.327976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:23.328075 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:23.327837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:23.328175 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.328083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:23.328175 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:23.328141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:25.327843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.327807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:25.327843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.327849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:25.328301 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.327852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:25.328301 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.327930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ftzw9" podUID="1f59ef6a-348f-4b5b-9831-e8f3038a4d1e"
Sep 16 00:34:25.328301 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.328032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9nw7" podUID="1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12"
Sep 16 00:34:25.328301 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.328110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2xsn" podUID="6539572a-35ca-4938-a691-21ff110ed047"
Sep 16 00:34:25.656930 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.656903    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-152.ec2.internal" event="NodeReady"
Sep 16 00:34:25.657117 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.657043    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 00:34:25.692604 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.692578    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:34:25.712723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.712696    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"]
Sep 16 00:34:25.712867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.712827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.716072 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.716044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 00:34:25.716880 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.716596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 00:34:25.716880 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.716763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 00:34:25.717166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.717103    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sptnt\""
Sep 16 00:34:25.723668 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.723629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 00:34:25.729029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.728834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"]
Sep 16 00:34:25.729029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.728967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"
Sep 16 00:34:25.731413 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.731395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4zzpm\""
Sep 16 00:34:25.732333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.732317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.732995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.732977    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.761184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.761152    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"]
Sep 16 00:34:25.761414 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.761391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.763794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.763768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-whctm\""
Sep 16 00:34:25.764049 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.764031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.764440 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.764423    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.764619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.764603    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 00:34:25.785157 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.785133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hzw72"]
Sep 16 00:34:25.785280 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.785261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:25.787868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.787833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.787868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.787859    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 00:34:25.788063 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.787899    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xpfr7\""
Sep 16 00:34:25.788063 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.787992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.788207 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.788194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 00:34:25.797319 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.797294    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-zxsw2"]
Sep 16 00:34:25.797445 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.797426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:25.800132 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.800109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.800375 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.800350    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 00:34:25.800375 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.800369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.800633 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.800602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8pshg\""
Sep 16 00:34:25.800730 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.800613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 00:34:25.805529 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.805508    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 00:34:25.811340 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.811321    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"]
Sep 16 00:34:25.811455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.811439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.813730 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.813716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.813819 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.813734    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 00:34:25.813819 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.813745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-msqgt\""
Sep 16 00:34:25.813929 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.813837    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 00:34:25.813929 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.813854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.819516 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.819488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 00:34:25.832446 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6cf\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd9g\" (UniqueName: \"kubernetes.io/projected/cad8511f-d5e6-46ee-820f-e699df814e7f-kube-api-access-zsd9g\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.832723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.832871 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82g8t\" (UniqueName: \"kubernetes.io/projected/e990c269-b651-4030-90df-1b8c31a5468d-kube-api-access-82g8t\") pod \"volume-data-source-validator-86695b7cc7-q9xwt\" (UID: \"e990c269-b651-4030-90df-1b8c31a5468d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"
Sep 16 00:34:25.832871 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.832871 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.832814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.833647 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:34:25.833682 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"]
Sep 16 00:34:25.833682 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833673    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"]
Sep 16 00:34:25.833742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"]
Sep 16 00:34:25.833742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hzw72"]
Sep 16 00:34:25.833742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-zxsw2"]
Sep 16 00:34:25.833742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833728    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"]
Sep 16 00:34:25.833742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.833732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:25.835810 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.835791    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 00:34:25.836035 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.836017    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.836477 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.836461    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 00:34:25.836805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.836785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.836858 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.836834    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l9n8m\""
Sep 16 00:34:25.847743 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.847719    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5qctk"]
Sep 16 00:34:25.847869 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.847853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:25.850363 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.850170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 00:34:25.850363 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.850232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.850363 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.850236    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 00:34:25.850363 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.850290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.850732 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.850707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vb54s\""
Sep 16 00:34:25.864035 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.864014    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"]
Sep 16 00:34:25.864176 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.864155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:25.867380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.866826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rpgmh\""
Sep 16 00:34:25.867380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.867267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 00:34:25.867636 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.867618    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 00:34:25.883406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.883357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:25.886314 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.886292    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5475d44bcb-9c4pt"]
Sep 16 00:34:25.886468 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.886449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:25.888691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.888671    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kmpkg\""
Sep 16 00:34:25.888797 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.888754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 00:34:25.888896 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.888880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 00:34:25.906393 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.906362    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bqqnb"]
Sep 16 00:34:25.906553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.906498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:25.909148 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z9xb4\""
Sep 16 00:34:25.909148 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909112    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 00:34:25.909348 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 00:34:25.909418 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909408    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.909602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.909694 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909599    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 00:34:25.909751 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.909703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 00:34:25.932442 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.932412    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"]
Sep 16 00:34:25.932598 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.932548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:25.933141 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4305e130-063f-4318-a07c-7a6cb748c1ac-serving-cert\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:25.933259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:25.933259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:25.933369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.933369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d826309-8374-4890-8ef0-bd8e1ba0e247-serving-cert\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:25.933369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7g6x\" (UniqueName: \"kubernetes.io/projected/4305e130-063f-4318-a07c-7a6cb748c1ac-kube-api-access-s7g6x\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:25.933515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6nm\" (UniqueName: \"kubernetes.io/projected/a633fe59-ba7f-4981-a1a0-0533327434e8-kube-api-access-4b6nm\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.933515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:25.933515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d826309-8374-4890-8ef0-bd8e1ba0e247-config\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:25.933515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82g8t\" (UniqueName: \"kubernetes.io/projected/e990c269-b651-4030-90df-1b8c31a5468d-kube-api-access-82g8t\") pod \"volume-data-source-validator-86695b7cc7-q9xwt\" (UID: \"e990c269-b651-4030-90df-1b8c31a5468d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-snapshots\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-service-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.933703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85b14f1-369a-4c29-9c33-91420ee78f85-tmp-dir\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7vm\" (UniqueName: \"kubernetes.io/projected/cc191bfb-a7ab-4ce4-994b-518fa5b494df-kube-api-access-gw7vm\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvsw\" (UniqueName: \"kubernetes.io/projected/b85b14f1-369a-4c29-9c33-91420ee78f85-kube-api-access-rbvsw\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.933861    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.933932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.433911027 +0000 UTC m=+36.595421291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:25.934064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.933934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.934373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-trusted-ca\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:25.934373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc191bfb-a7ab-4ce4-994b-518fa5b494df-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:25.934373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgp9\" (UniqueName: \"kubernetes.io/projected/3d826309-8374-4890-8ef0-bd8e1ba0e247-kube-api-access-7sgp9\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:25.934373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6cf\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.934373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ljm\" (UniqueName: \"kubernetes.io/projected/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-kube-api-access-n6ljm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:25.934619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b85b14f1-369a-4c29-9c33-91420ee78f85-config-volume\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:25.934619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.934619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.934619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd9g\" (UniqueName: \"kubernetes.io/projected/cad8511f-d5e6-46ee-820f-e699df814e7f-kube-api-access-zsd9g\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.934822 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-config\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:25.934822 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-tmp\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.934822 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.934822 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a633fe59-ba7f-4981-a1a0-0533327434e8-serving-cert\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.935039 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.935039 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.935039 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.934904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:25.935194 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.935120    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:25.935194 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.935147    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:25.935194 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.935155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.935282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.935211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.935282 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:25.935240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.435224434 +0000 UTC m=+36.596734699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:25.935616 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.935430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 00:34:25.935616 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.935491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cs9kq\""
Sep 16 00:34:25.939575 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.939553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.939765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.939741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.941496 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.941477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.942872 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.942825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6cf\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:25.943128 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.943095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd9g\" (UniqueName: \"kubernetes.io/projected/cad8511f-d5e6-46ee-820f-e699df814e7f-kube-api-access-zsd9g\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:25.943635 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.943609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82g8t\" (UniqueName: \"kubernetes.io/projected/e990c269-b651-4030-90df-1b8c31a5468d-kube-api-access-82g8t\") pod \"volume-data-source-validator-86695b7cc7-q9xwt\" (UID: \"e990c269-b651-4030-90df-1b8c31a5468d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"
Sep 16 00:34:25.952913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.952890    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"]
Sep 16 00:34:25.953036 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.952939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qctk"]
Sep 16 00:34:25.953036 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953000    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqqnb"]
Sep 16 00:34:25.953036 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"]
Sep 16 00:34:25.953181 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"]
Sep 16 00:34:25.953181 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"
Sep 16 00:34:25.953181 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5475d44bcb-9c4pt"]
Sep 16 00:34:25.953181 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.953090    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"]
Sep 16 00:34:25.955410 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.955387    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 00:34:25.955511 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.955421    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 00:34:25.955511 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.955419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wrpd2\""
Sep 16 00:34:25.971666 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:25.971635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:26.035345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-snapshots\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.035345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-service-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85b14f1-369a-4c29-9c33-91420ee78f85-tmp-dir\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/665ca7f3-d81e-4286-b5ce-1074c872f927-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7vm\" (UniqueName: \"kubernetes.io/projected/cc191bfb-a7ab-4ce4-994b-518fa5b494df-kube-api-access-gw7vm\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.035539    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:26.035579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvsw\" (UniqueName: \"kubernetes.io/projected/b85b14f1-369a-4c29-9c33-91420ee78f85-kube-api-access-rbvsw\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.035609    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.535587468 +0000 UTC m=+36.697097718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-trusted-ca\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc191bfb-a7ab-4ce4-994b-518fa5b494df-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgp9\" (UniqueName: \"kubernetes.io/projected/3d826309-8374-4890-8ef0-bd8e1ba0e247-kube-api-access-7sgp9\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ljm\" (UniqueName: \"kubernetes.io/projected/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-kube-api-access-n6ljm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85b14f1-369a-4c29-9c33-91420ee78f85-tmp-dir\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b85b14f1-369a-4c29-9c33-91420ee78f85-config-volume\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.035855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.035823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzjw\" (UniqueName: \"kubernetes.io/projected/faeedb1b-33ae-45f8-80fe-90ea7ee2f866-kube-api-access-fhzjw\") pod \"network-check-source-6f8b768455-bzbwp\" (UID: \"faeedb1b-33ae-45f8-80fe-90ea7ee2f866\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-config\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-tmp\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-stats-auth\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a633fe59-ba7f-4981-a1a0-0533327434e8-serving-cert\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628c8\" (UniqueName: \"kubernetes.io/projected/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-kube-api-access-628c8\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-default-certificate\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-snapshots\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4305e130-063f-4318-a07c-7a6cb748c1ac-serving-cert\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d826309-8374-4890-8ef0-bd8e1ba0e247-serving-cert\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b85b14f1-369a-4c29-9c33-91420ee78f85-config-volume\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7g6x\" (UniqueName: \"kubernetes.io/projected/4305e130-063f-4318-a07c-7a6cb748c1ac-kube-api-access-s7g6x\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a739a79-a67d-4add-abca-aa30d2f17687-cert\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.036481    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6nm\" (UniqueName: \"kubernetes.io/projected/a633fe59-ba7f-4981-a1a0-0533327434e8-kube-api-access-4b6nm\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc191bfb-a7ab-4ce4-994b-518fa5b494df-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zw5\" (UniqueName: \"kubernetes.io/projected/2a739a79-a67d-4add-abca-aa30d2f17687-kube-api-access-w6zw5\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.036542    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.536526709 +0000 UTC m=+36.698036960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a633fe59-ba7f-4981-a1a0-0533327434e8-tmp\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-config\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.036775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4305e130-063f-4318-a07c-7a6cb748c1ac-trusted-ca\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.037691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-service-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.037691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.037691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.036993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d826309-8374-4890-8ef0-bd8e1ba0e247-config\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.037691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.037485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a633fe59-ba7f-4981-a1a0-0533327434e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.037691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.037685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d826309-8374-4890-8ef0-bd8e1ba0e247-config\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.037943 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.037838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.038868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.038843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4305e130-063f-4318-a07c-7a6cb748c1ac-serving-cert\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.039292 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.039269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d826309-8374-4890-8ef0-bd8e1ba0e247-serving-cert\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.039414 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.039393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a633fe59-ba7f-4981-a1a0-0533327434e8-serving-cert\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.039454 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.039422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.045211 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.045184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgp9\" (UniqueName: \"kubernetes.io/projected/3d826309-8374-4890-8ef0-bd8e1ba0e247-kube-api-access-7sgp9\") pod \"service-ca-operator-cf7c776b7-9kxj9\" (UID: \"3d826309-8374-4890-8ef0-bd8e1ba0e247\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.045370 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.045347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"
Sep 16 00:34:26.045767 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.045744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7g6x\" (UniqueName: \"kubernetes.io/projected/4305e130-063f-4318-a07c-7a6cb748c1ac-kube-api-access-s7g6x\") pod \"console-operator-696884fb9c-hzw72\" (UID: \"4305e130-063f-4318-a07c-7a6cb748c1ac\") " pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.045842 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.045792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7vm\" (UniqueName: \"kubernetes.io/projected/cc191bfb-a7ab-4ce4-994b-518fa5b494df-kube-api-access-gw7vm\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.046051 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.046021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ljm\" (UniqueName: \"kubernetes.io/projected/7cde447c-7ce6-4fa6-98a2-4853d9fd5179-kube-api-access-n6ljm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5dc8v\" (UID: \"7cde447c-7ce6-4fa6-98a2-4853d9fd5179\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.046430 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.046414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvsw\" (UniqueName: \"kubernetes.io/projected/b85b14f1-369a-4c29-9c33-91420ee78f85-kube-api-access-rbvsw\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.047216 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.047199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6nm\" (UniqueName: \"kubernetes.io/projected/a633fe59-ba7f-4981-a1a0-0533327434e8-kube-api-access-4b6nm\") pod \"insights-operator-5844b856db-zxsw2\" (UID: \"a633fe59-ba7f-4981-a1a0-0533327434e8\") " pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.121407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.121376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"
Sep 16 00:34:26.127510 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.127487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:26.131609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.131593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-zxsw2"
Sep 16 00:34:26.137975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.137938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzjw\" (UniqueName: \"kubernetes.io/projected/faeedb1b-33ae-45f8-80fe-90ea7ee2f866-kube-api-access-fhzjw\") pod \"network-check-source-6f8b768455-bzbwp\" (UID: \"faeedb1b-33ae-45f8-80fe-90ea7ee2f866\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"
Sep 16 00:34:26.138126 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.137982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.138126 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.138101    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.638087003 +0000 UTC m=+36.799597253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:26.138126 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-stats-auth\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.138283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628c8\" (UniqueName: \"kubernetes.io/projected/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-kube-api-access-628c8\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.138283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-default-certificate\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.138374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a739a79-a67d-4add-abca-aa30d2f17687-cert\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.138374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zw5\" (UniqueName: \"kubernetes.io/projected/2a739a79-a67d-4add-abca-aa30d2f17687-kube-api-access-w6zw5\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.138374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.138520 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/665ca7f3-d81e-4286-b5ce-1074c872f927-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.138520 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.138469    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:26.138520 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.138474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.138678 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.138528    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.638513159 +0000 UTC m=+36.800023411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:26.138678 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.138577    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:26.138678 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.138619    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:26.638605368 +0000 UTC m=+36.800115622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:26.139190 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.139113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/665ca7f3-d81e-4286-b5ce-1074c872f927-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.140944 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.140919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a739a79-a67d-4add-abca-aa30d2f17687-cert\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.141091 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.141011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-default-certificate\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.141170 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.141121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-stats-auth\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.148381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.148278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzjw\" (UniqueName: \"kubernetes.io/projected/faeedb1b-33ae-45f8-80fe-90ea7ee2f866-kube-api-access-fhzjw\") pod \"network-check-source-6f8b768455-bzbwp\" (UID: \"faeedb1b-33ae-45f8-80fe-90ea7ee2f866\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"
Sep 16 00:34:26.149341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.149295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628c8\" (UniqueName: \"kubernetes.io/projected/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-kube-api-access-628c8\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.151347 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.150311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zw5\" (UniqueName: \"kubernetes.io/projected/2a739a79-a67d-4add-abca-aa30d2f17687-kube-api-access-w6zw5\") pod \"ingress-canary-bqqnb\" (UID: \"2a739a79-a67d-4add-abca-aa30d2f17687\") " pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.157590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.157436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"
Sep 16 00:34:26.256261 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.256236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bqqnb"
Sep 16 00:34:26.301902 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.301867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"
Sep 16 00:34:26.367530 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.367500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt"]
Sep 16 00:34:26.382650 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.382621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode990c269_b651_4030_90df_1b8c31a5468d.slice/crio-bceb09ee34615bf3e314c546af77c90cbdd2abd8899040d49761b68a6bc0c2af WatchSource:0}: Error finding container bceb09ee34615bf3e314c546af77c90cbdd2abd8899040d49761b68a6bc0c2af: Status 404 returned error can't find the container with id bceb09ee34615bf3e314c546af77c90cbdd2abd8899040d49761b68a6bc0c2af
Sep 16 00:34:26.430294 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.430218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v"]
Sep 16 00:34:26.433666 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.433641    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cde447c_7ce6_4fa6_98a2_4853d9fd5179.slice/crio-16d9f459eb76b49e77e480715d6ef29ec310397d68d601d925c87e99ce9c9a1c WatchSource:0}: Error finding container 16d9f459eb76b49e77e480715d6ef29ec310397d68d601d925c87e99ce9c9a1c: Status 404 returned error can't find the container with id 16d9f459eb76b49e77e480715d6ef29ec310397d68d601d925c87e99ce9c9a1c
Sep 16 00:34:26.442425 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.442402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:26.442533 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.442474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:26.442594 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.442544    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:26.442594 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.442564    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:26.442690 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.442624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.442604646 +0000 UTC m=+37.604114896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:26.442690 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.442632    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:26.442783 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.442703    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.442685707 +0000 UTC m=+37.604195970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:26.459033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.459008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hzw72"]
Sep 16 00:34:26.461548 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.461522    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4305e130_063f_4318_a07c_7a6cb748c1ac.slice/crio-577723cf5bc7c2891fb4280f656f4ab4947b0cbaf2b365eaec13e0eb7b2dee52 WatchSource:0}: Error finding container 577723cf5bc7c2891fb4280f656f4ab4947b0cbaf2b365eaec13e0eb7b2dee52: Status 404 returned error can't find the container with id 577723cf5bc7c2891fb4280f656f4ab4947b0cbaf2b365eaec13e0eb7b2dee52
Sep 16 00:34:26.473371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.473344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9"]
Sep 16 00:34:26.476246 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.476223    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d826309_8374_4890_8ef0_bd8e1ba0e247.slice/crio-4747a80e0912e94a065b2dc5b1852eecfaac6cbdb6ecbe8fab5356ae63267984 WatchSource:0}: Error finding container 4747a80e0912e94a065b2dc5b1852eecfaac6cbdb6ecbe8fab5356ae63267984: Status 404 returned error can't find the container with id 4747a80e0912e94a065b2dc5b1852eecfaac6cbdb6ecbe8fab5356ae63267984
Sep 16 00:34:26.493208 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.493182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-zxsw2"]
Sep 16 00:34:26.495811 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.495789    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda633fe59_ba7f_4981_a1a0_0533327434e8.slice/crio-daf6f08436afdc4cd2109b8bb14469738a0a442083958555f9ea554556ee37ed WatchSource:0}: Error finding container daf6f08436afdc4cd2109b8bb14469738a0a442083958555f9ea554556ee37ed: Status 404 returned error can't find the container with id daf6f08436afdc4cd2109b8bb14469738a0a442083958555f9ea554556ee37ed
Sep 16 00:34:26.543323 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.543293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:26.543471 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.543405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:26.543471 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.543456    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:26.543550 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.543528    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:26.543550 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.543540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.543523851 +0000 UTC m=+37.705034101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:26.543617 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.543576    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.543564446 +0000 UTC m=+37.705074696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:26.549612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.549478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bqqnb"]
Sep 16 00:34:26.552124 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.552079    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a739a79_a67d_4add_abca_aa30d2f17687.slice/crio-887136608fd0b2ec216c0e5e96c17f04cc948171386c257005d5e007899cb956 WatchSource:0}: Error finding container 887136608fd0b2ec216c0e5e96c17f04cc948171386c257005d5e007899cb956: Status 404 returned error can't find the container with id 887136608fd0b2ec216c0e5e96c17f04cc948171386c257005d5e007899cb956
Sep 16 00:34:26.602908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.602866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp"]
Sep 16 00:34:26.604241 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.604203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" event={"ID":"4305e130-063f-4318-a07c-7a6cb748c1ac","Type":"ContainerStarted","Data":"577723cf5bc7c2891fb4280f656f4ab4947b0cbaf2b365eaec13e0eb7b2dee52"}
Sep 16 00:34:26.605336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.605307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v" event={"ID":"7cde447c-7ce6-4fa6-98a2-4853d9fd5179","Type":"ContainerStarted","Data":"16d9f459eb76b49e77e480715d6ef29ec310397d68d601d925c87e99ce9c9a1c"}
Sep 16 00:34:26.606167 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:26.606142    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeedb1b_33ae_45f8_80fe_90ea7ee2f866.slice/crio-aaa94123c05c868f1a35dc048d69170ebf0f87c9df4cd970832a43c0f4ff2dc3 WatchSource:0}: Error finding container aaa94123c05c868f1a35dc048d69170ebf0f87c9df4cd970832a43c0f4ff2dc3: Status 404 returned error can't find the container with id aaa94123c05c868f1a35dc048d69170ebf0f87c9df4cd970832a43c0f4ff2dc3
Sep 16 00:34:26.606532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.606503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-zxsw2" event={"ID":"a633fe59-ba7f-4981-a1a0-0533327434e8","Type":"ContainerStarted","Data":"daf6f08436afdc4cd2109b8bb14469738a0a442083958555f9ea554556ee37ed"}
Sep 16 00:34:26.607649 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.607628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqqnb" event={"ID":"2a739a79-a67d-4add-abca-aa30d2f17687","Type":"ContainerStarted","Data":"887136608fd0b2ec216c0e5e96c17f04cc948171386c257005d5e007899cb956"}
Sep 16 00:34:26.608739 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.608717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt" event={"ID":"e990c269-b651-4030-90df-1b8c31a5468d","Type":"ContainerStarted","Data":"bceb09ee34615bf3e314c546af77c90cbdd2abd8899040d49761b68a6bc0c2af"}
Sep 16 00:34:26.611524 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.611504    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="f21834de08149626dd77df47700307f43a4170569b75d3b7a4892657ecdc4a5e" exitCode=0
Sep 16 00:34:26.611604 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.611570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"f21834de08149626dd77df47700307f43a4170569b75d3b7a4892657ecdc4a5e"}
Sep 16 00:34:26.612647 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.612629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9" event={"ID":"3d826309-8374-4890-8ef0-bd8e1ba0e247","Type":"ContainerStarted","Data":"4747a80e0912e94a065b2dc5b1852eecfaac6cbdb6ecbe8fab5356ae63267984"}
Sep 16 00:34:26.644015 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.643974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.644212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.644080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:26.644212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:26.644130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:26.644212 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.644165    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.644140054 +0000 UTC m=+37.805650305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:26.644388 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.644218    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:26.644388 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.644231    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:26.644388 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.644269    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.644258086 +0000 UTC m=+37.805768336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:26.644388 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:26.644292    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:27.64427554 +0000 UTC m=+37.805785810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:27.328334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.327359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:27.328334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.327849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:27.328334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.328035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:27.332859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.332540    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:34:27.332859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.332666    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7jvk\""
Sep 16 00:34:27.335231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.334703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:34:27.335231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.334989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgg7z\""
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.452045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.452176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.452371    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.452437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.45241675 +0000 UTC m=+39.613927016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.453157    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.453173    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:27.453256 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.453220    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.453204661 +0000 UTC m=+39.614714911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:27.554289 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.554247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:27.554485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.554426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:27.555245 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.554623    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:27.555245 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.554698    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.554679027 +0000 UTC m=+39.716189282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:27.555245 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.554785    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:27.555245 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.554825    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.5548122 +0000 UTC m=+39.716322456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:27.625747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.624734    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a72feb0-e226-4441-b535-e80f772ad304" containerID="4f170ca8245a15abd404c9b43e32d76b22d017f522cfc73ca042c28a06fe9839" exitCode=0
Sep 16 00:34:27.625747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.624829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerDied","Data":"4f170ca8245a15abd404c9b43e32d76b22d017f522cfc73ca042c28a06fe9839"}
Sep 16 00:34:27.631349 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.631309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp" event={"ID":"faeedb1b-33ae-45f8-80fe-90ea7ee2f866","Type":"ContainerStarted","Data":"aaa94123c05c868f1a35dc048d69170ebf0f87c9df4cd970832a43c0f4ff2dc3"}
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.655738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.655835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:27.655891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.656353    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.656446    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.656497    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.656080676 +0000 UTC m=+39.817590950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.656516    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.656504949 +0000 UTC m=+39.818015206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:27.656724 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:27.656531    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:29.656522582 +0000 UTC m=+39.818032838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:28.645985 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:28.645311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gkht" event={"ID":"0a72feb0-e226-4441-b535-e80f772ad304","Type":"ContainerStarted","Data":"4e768d9314841dfc5b3df927c0f9ba3e8f74e06d3bc44ed64c9aa421be8c9469"}
Sep 16 00:34:28.793529 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:28.793363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:28.849132 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:28.849094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:29.478519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.478268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:29.478519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.478426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:29.478519 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.478464    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:29.478519 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.478489    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:29.478826 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.478559    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.478541088 +0000 UTC m=+43.640051338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:29.478826 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.478593    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:29.478826 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.478666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.478647647 +0000 UTC m=+43.640157903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:29.580030 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.579827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:29.580030 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.579937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:29.580030 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.580031    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:29.580273 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.580063    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:29.580273 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.580105    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.580082799 +0000 UTC m=+43.741593049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:29.580273 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.580127    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.580117549 +0000 UTC m=+43.741627804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:29.680996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.680889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.681000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.681071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.681084    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.681168    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.681145181 +0000 UTC m=+43.842655452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.681221    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.681203973 +0000 UTC m=+43.842714236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.681313    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:29.681479 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:29.681348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:33.681336296 +0000 UTC m=+43.842846577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:29.831560 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.831518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:29.932808 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:29.932713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log" limit=1024
Sep 16 00:34:33.517264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.517216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.517343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.517396    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.517415    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.517485    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.517464558 +0000 UTC m=+51.678974809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.517523    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:33.517734 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.517601    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.517582768 +0000 UTC m=+51.679093035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:33.618469 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.618420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:33.618647 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.618550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:33.618647 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.618576    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:33.618647 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.618639    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.618622773 +0000 UTC m=+51.780133023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:33.618775 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.618701    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:33.618775 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.618759    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.618745869 +0000 UTC m=+51.780256121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:33.719777 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.719738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:33.719939 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.719807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:33.719939 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:33.719852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:33.719939 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.719891    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:33.720072 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.719980    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.719944013 +0000 UTC m=+51.881454263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:33.720072 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.719992    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:33.720072 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.719999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.719990972 +0000 UTC m=+51.881501223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:33.720072 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:33.720064    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:41.720048365 +0000 UTC m=+51.881558615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:34.665813 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.665768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-zxsw2" event={"ID":"a633fe59-ba7f-4981-a1a0-0533327434e8","Type":"ContainerStarted","Data":"2d0f68766e4bb9d89e6ba43bf75a35e55c277bb7edca50607a15b34f01499b1b"}
Sep 16 00:34:34.668045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.668019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bqqnb" event={"ID":"2a739a79-a67d-4add-abca-aa30d2f17687","Type":"ContainerStarted","Data":"294f70b62c67e69b2d487716f510da8f90e0c3c12b4881126c968258bf3e124d"}
Sep 16 00:34:34.670190 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.670165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt" event={"ID":"e990c269-b651-4030-90df-1b8c31a5468d","Type":"ContainerStarted","Data":"f7a8ff8d75590c9238e84012040246a52fe5f136493c0f58f753085228028094"}
Sep 16 00:34:34.672060 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.672036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9" event={"ID":"3d826309-8374-4890-8ef0-bd8e1ba0e247","Type":"ContainerStarted","Data":"69fe8edca3c3adbe6ab53e1331b64b8eae54c01e02d6059a5e992611bdf92acc"}
Sep 16 00:34:34.674313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.674294    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/0.log"
Sep 16 00:34:34.674366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.674331    2577 generic.go:356] "Generic (PLEG): container finished" podID="4305e130-063f-4318-a07c-7a6cb748c1ac" containerID="80e509c22a60e10ce867542ef7087b1e3dd266c127e9447e0f09d4c6733f1f9a" exitCode=255
Sep 16 00:34:34.674430 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.674396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" event={"ID":"4305e130-063f-4318-a07c-7a6cb748c1ac","Type":"ContainerDied","Data":"80e509c22a60e10ce867542ef7087b1e3dd266c127e9447e0f09d4c6733f1f9a"}
Sep 16 00:34:34.674666 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.674651    2577 scope.go:117] "RemoveContainer" containerID="80e509c22a60e10ce867542ef7087b1e3dd266c127e9447e0f09d4c6733f1f9a"
Sep 16 00:34:34.677278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.677253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v" event={"ID":"7cde447c-7ce6-4fa6-98a2-4853d9fd5179","Type":"ContainerStarted","Data":"8237a79dd4177579aeac442daa04a47b68500919878e31eaea895e27b6fb7240"}
Sep 16 00:34:34.679195 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.679172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp" event={"ID":"faeedb1b-33ae-45f8-80fe-90ea7ee2f866","Type":"ContainerStarted","Data":"0ff4d2dedbc5bc8a017ed8d94ae762a0a9767652d7a925571acdc82abdb61be7"}
Sep 16 00:34:34.681560 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.681513    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5gkht" podStartSLOduration=11.616137969 podStartE2EDuration="43.681497761s" totalImagesPullingTime="32.065359792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:28.669793214 +0000 UTC m=+38.831303488" watchObservedRunningTime="2026-09-16 00:34:34.681497761 +0000 UTC m=+44.843008034"
Sep 16 00:34:34.702661 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.702599    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v" podStartSLOduration=15.887393125 podStartE2EDuration="23.702577779s" totalImagesPullingTime="7.815184654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.702515604 +0000 UTC m=+44.864025875" watchObservedRunningTime="2026-09-16 00:34:34.702577779 +0000 UTC m=+44.864088047"
Sep 16 00:34:34.703990 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.703932    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-zxsw2" podStartSLOduration=15.950256605 podStartE2EDuration="23.703914117s" totalImagesPullingTime="7.753657512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.682407534 +0000 UTC m=+44.843917802" watchObservedRunningTime="2026-09-16 00:34:34.703914117 +0000 UTC m=+44.865424391"
Sep 16 00:34:34.720810 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.720742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bqqnb" podStartSLOduration=2.023408274 podStartE2EDuration="9.720720447s" totalImagesPullingTime="7.697312173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.720199967 +0000 UTC m=+44.881710259" watchObservedRunningTime="2026-09-16 00:34:34.720720447 +0000 UTC m=+44.882230721"
Sep 16 00:34:34.738802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.736449    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bzbwp" podStartSLOduration=16.082974475 podStartE2EDuration="23.73641102s" totalImagesPullingTime="7.653436545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.735699836 +0000 UTC m=+44.897210110" watchObservedRunningTime="2026-09-16 00:34:34.73641102 +0000 UTC m=+44.897921293"
Sep 16 00:34:34.769641 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.769575    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q9xwt" podStartSLOduration=16.037113831 podStartE2EDuration="23.769554684s" totalImagesPullingTime="7.732440853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.76898296 +0000 UTC m=+44.930493230" watchObservedRunningTime="2026-09-16 00:34:34.769554684 +0000 UTC m=+44.931064949"
Sep 16 00:34:34.791639 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:34.790745    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9" podStartSLOduration=16.019128942 podStartE2EDuration="23.79069247s" totalImagesPullingTime="7.771563528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:34.788750219 +0000 UTC m=+44.950260492" watchObservedRunningTime="2026-09-16 00:34:34.79069247 +0000 UTC m=+44.952202743"
Sep 16 00:34:35.693025 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.692924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:34:35.693494 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.693481    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/0.log"
Sep 16 00:34:35.693566 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.693526    2577 generic.go:356] "Generic (PLEG): container finished" podID="4305e130-063f-4318-a07c-7a6cb748c1ac" containerID="2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e" exitCode=255
Sep 16 00:34:35.693634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.693615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" event={"ID":"4305e130-063f-4318-a07c-7a6cb748c1ac","Type":"ContainerDied","Data":"2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e"}
Sep 16 00:34:35.693712 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.693700    2577 scope.go:117] "RemoveContainer" containerID="80e509c22a60e10ce867542ef7087b1e3dd266c127e9447e0f09d4c6733f1f9a"
Sep 16 00:34:35.695232 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:35.695008    2577 scope.go:117] "RemoveContainer" containerID="2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e"
Sep 16 00:34:35.695232 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:35.695157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hzw72_openshift-console-operator(4305e130-063f-4318-a07c-7a6cb748c1ac)\"" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" podUID="4305e130-063f-4318-a07c-7a6cb748c1ac"
Sep 16 00:34:36.128176 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.128135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:36.128462 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.128202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:36.351747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.351712    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"]
Sep 16 00:34:36.371856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.371825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"]
Sep 16 00:34:36.372063 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.371943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"
Sep 16 00:34:36.374841 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.374807    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 00:34:36.375037 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.374809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 00:34:36.376012 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.375937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jr9vh\""
Sep 16 00:34:36.449371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.449329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2z29\" (UniqueName: \"kubernetes.io/projected/dcbeaf8f-1b56-4ccb-9254-c580e53589d8-kube-api-access-t2z29\") pod \"migrator-8b4d697c5-45fpt\" (UID: \"dcbeaf8f-1b56-4ccb-9254-c580e53589d8\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"
Sep 16 00:34:36.551085 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.551042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2z29\" (UniqueName: \"kubernetes.io/projected/dcbeaf8f-1b56-4ccb-9254-c580e53589d8-kube-api-access-t2z29\") pod \"migrator-8b4d697c5-45fpt\" (UID: \"dcbeaf8f-1b56-4ccb-9254-c580e53589d8\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"
Sep 16 00:34:36.560906 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.560846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2z29\" (UniqueName: \"kubernetes.io/projected/dcbeaf8f-1b56-4ccb-9254-c580e53589d8-kube-api-access-t2z29\") pod \"migrator-8b4d697c5-45fpt\" (UID: \"dcbeaf8f-1b56-4ccb-9254-c580e53589d8\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"
Sep 16 00:34:36.620167 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.620133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hdpd7"]
Sep 16 00:34:36.657669 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.657591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdpd7"]
Sep 16 00:34:36.657808 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.657720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.660392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.660370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 00:34:36.660530 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.660369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 00:34:36.660804 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.660787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4hl7\""
Sep 16 00:34:36.684912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.684885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"
Sep 16 00:34:36.698785 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.698761    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:34:36.699162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.699105    2577 scope.go:117] "RemoveContainer" containerID="2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e"
Sep 16 00:34:36.699275 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:36.699256    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hzw72_openshift-console-operator(4305e130-063f-4318-a07c-7a6cb748c1ac)\"" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" podUID="4305e130-063f-4318-a07c-7a6cb748c1ac"
Sep 16 00:34:36.754601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.753885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-data-volume\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.754601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.753936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.754601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.754042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7jw\" (UniqueName: \"kubernetes.io/projected/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-api-access-sd7jw\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.754601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.754123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-crio-socket\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.754601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.754167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.854854 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.854815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-crio-socket\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.854854 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.854860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.855103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.854939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-data-volume\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.855103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.855038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.855103 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:36.855072    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:36.855223 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.855099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-crio-socket\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.855223 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.855115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7jw\" (UniqueName: \"kubernetes.io/projected/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-api-access-sd7jw\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.855223 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:36.855154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:37.355133814 +0000 UTC m=+47.516644084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:36.855223 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:36.855157    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:34:36.855223 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:36.855215    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:37.355198682 +0000 UTC m=+47.516708952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : configmap "kube-rbac-proxy" not found
Sep 16 00:34:36.855398 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.855286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-data-volume\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.863411 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.863381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7jw\" (UniqueName: \"kubernetes.io/projected/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-api-access-sd7jw\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:36.973571 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:36.973408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt"]
Sep 16 00:34:36.976028 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:36.975996    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbeaf8f_1b56_4ccb_9254_c580e53589d8.slice/crio-5906f0c1dc81f48f914c9220ccda735e7c663ae1cb7f9da97555fd5f930059c8 WatchSource:0}: Error finding container 5906f0c1dc81f48f914c9220ccda735e7c663ae1cb7f9da97555fd5f930059c8: Status 404 returned error can't find the container with id 5906f0c1dc81f48f914c9220ccda735e7c663ae1cb7f9da97555fd5f930059c8
Sep 16 00:34:37.359931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:37.359893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:37.360125 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:37.360024    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:34:37.360125 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:37.360030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:37.360125 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:37.360094    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:38.360074206 +0000 UTC m=+48.521584537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : configmap "kube-rbac-proxy" not found
Sep 16 00:34:37.360243 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:37.360178    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:37.360276 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:37.360240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:38.360225012 +0000 UTC m=+48.521735314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:37.705000 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:37.704930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt" event={"ID":"dcbeaf8f-1b56-4ccb-9254-c580e53589d8","Type":"ContainerStarted","Data":"5906f0c1dc81f48f914c9220ccda735e7c663ae1cb7f9da97555fd5f930059c8"}
Sep 16 00:34:37.705437 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:37.705304    2577 scope.go:117] "RemoveContainer" containerID="2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e"
Sep 16 00:34:37.705564 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:37.705539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hzw72_openshift-console-operator(4305e130-063f-4318-a07c-7a6cb748c1ac)\"" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" podUID="4305e130-063f-4318-a07c-7a6cb748c1ac"
Sep 16 00:34:38.359388 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.359351    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vwcfg"]
Sep 16 00:34:38.369698 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.369666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:38.369877 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.369772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:38.369877 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:38.369826    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:34:38.369877 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:38.369879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:40.369864737 +0000 UTC m=+50.531374987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : configmap "kube-rbac-proxy" not found
Sep 16 00:34:38.370077 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:38.369971    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:38.370077 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:38.370047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:40.370026321 +0000 UTC m=+50.531536591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:38.379493 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.379469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vwcfg"]
Sep 16 00:34:38.379618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.379558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.382059 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.382030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 00:34:38.382230 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.382128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 00:34:38.382308 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.382230    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8jkt2\""
Sep 16 00:34:38.383245 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.383206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 00:34:38.383370 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.383278    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 00:34:38.470336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.470295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/db70735a-5712-4a61-9357-9301d7bb5996-signing-key\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.470473 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.470352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/db70735a-5712-4a61-9357-9301d7bb5996-signing-cabundle\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.470473 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.470416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqkg\" (UniqueName: \"kubernetes.io/projected/db70735a-5712-4a61-9357-9301d7bb5996-kube-api-access-lkqkg\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.571407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.571380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/db70735a-5712-4a61-9357-9301d7bb5996-signing-key\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.571477 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.571438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/db70735a-5712-4a61-9357-9301d7bb5996-signing-cabundle\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.571477 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.571470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqkg\" (UniqueName: \"kubernetes.io/projected/db70735a-5712-4a61-9357-9301d7bb5996-kube-api-access-lkqkg\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.572140 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.572117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/db70735a-5712-4a61-9357-9301d7bb5996-signing-cabundle\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.574021 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.574001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/db70735a-5712-4a61-9357-9301d7bb5996-signing-key\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.579419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.579393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqkg\" (UniqueName: \"kubernetes.io/projected/db70735a-5712-4a61-9357-9301d7bb5996-kube-api-access-lkqkg\") pod \"service-ca-568869fdc5-vwcfg\" (UID: \"db70735a-5712-4a61-9357-9301d7bb5996\") " pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.688278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.688253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-vwcfg"
Sep 16 00:34:38.713625 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.713593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt" event={"ID":"dcbeaf8f-1b56-4ccb-9254-c580e53589d8","Type":"ContainerStarted","Data":"60d3e2c0c82ebfe95aeaf234578244a8f2c6041ac02fe88b65208ef8362dcd97"}
Sep 16 00:34:38.817508 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.817482    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzgkz_3226ed81-5d7e-4d58-837d-99ede28bbcbf/dns-node-resolver/0.log"
Sep 16 00:34:38.973931 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:38.973905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vwcfg"]
Sep 16 00:34:38.974932 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:38.974909    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb70735a_5712_4a61_9357_9301d7bb5996.slice/crio-6da9585d9f2d99590a0893f31e703cb0143b5014c2dd607800af2d072812994a WatchSource:0}: Error finding container 6da9585d9f2d99590a0893f31e703cb0143b5014c2dd607800af2d072812994a: Status 404 returned error can't find the container with id 6da9585d9f2d99590a0893f31e703cb0143b5014c2dd607800af2d072812994a
Sep 16 00:34:39.719901 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.719865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-vwcfg" event={"ID":"db70735a-5712-4a61-9357-9301d7bb5996","Type":"ContainerStarted","Data":"0f4e10d7466ca77969a8ccf6118540a2b451d51c51afba4a8a322283320d0401"}
Sep 16 00:34:39.720338 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.719925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-vwcfg" event={"ID":"db70735a-5712-4a61-9357-9301d7bb5996","Type":"ContainerStarted","Data":"6da9585d9f2d99590a0893f31e703cb0143b5014c2dd607800af2d072812994a"}
Sep 16 00:34:39.721932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.721901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt" event={"ID":"dcbeaf8f-1b56-4ccb-9254-c580e53589d8","Type":"ContainerStarted","Data":"737b85bb8dd161814dc069b8379e79df8ea28db6e9c6ad00f15d0b8e1ac393d2"}
Sep 16 00:34:39.740393 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.740333    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-vwcfg" podStartSLOduration=1.740311029 podStartE2EDuration="1.740311029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:39.738973004 +0000 UTC m=+49.900483273" watchObservedRunningTime="2026-09-16 00:34:39.740311029 +0000 UTC m=+49.901821302"
Sep 16 00:34:39.756039 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.755992    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-45fpt" podStartSLOduration=2.190123899 podStartE2EDuration="3.755978938s" totalImagesPullingTime="1.565855039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:39.755011062 +0000 UTC m=+49.916521337" watchObservedRunningTime="2026-09-16 00:34:39.755978938 +0000 UTC m=+49.917489221"
Sep 16 00:34:39.812291 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:39.812264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztd8m_9b83e392-9d9a-477b-9d05-cd1b5d124150/node-ca/0.log"
Sep 16 00:34:40.212001 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.211926    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqqnb_2a739a79-a67d-4add-abca-aa30d2f17687/serve-healthcheck-canary/0.log"
Sep 16 00:34:40.388170 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.388137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:40.388366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.388276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:40.388366 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:40.388296    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:40.388366 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:40.388357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:44.388339793 +0000 UTC m=+54.549850063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:40.388366 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:40.388359    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 00:34:40.388591 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:40.388395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:44.388385394 +0000 UTC m=+54.549895656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : configmap "kube-rbac-proxy" not found
Sep 16 00:34:40.413382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.413350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-45fpt_dcbeaf8f-1b56-4ccb-9254-c580e53589d8/migrator/0.log"
Sep 16 00:34:40.616450 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.616421    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-45fpt_dcbeaf8f-1b56-4ccb-9254-c580e53589d8/graceful-termination/0.log"
Sep 16 00:34:40.819048 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:40.819020    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5dc8v_7cde447c-7ce6-4fa6-98a2-4853d9fd5179/kube-storage-version-migrator-operator/0.log"
Sep 16 00:34:41.601634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.601591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:41.601862 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.601709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:41.601862 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.601783    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 00:34:41.601862 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.601807    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5948fb99cf-nl9jx: secret "image-registry-tls" not found
Sep 16 00:34:41.601862 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.601852    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 00:34:41.602109 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.601880    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls podName:4cf050e9-b781-4881-b040-104f2fa91926 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.601856877 +0000 UTC m=+67.763367145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls") pod "image-registry-5948fb99cf-nl9jx" (UID: "4cf050e9-b781-4881-b040-104f2fa91926") : secret "image-registry-tls" not found
Sep 16 00:34:41.602109 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.601911    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls podName:cad8511f-d5e6-46ee-820f-e699df814e7f nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.60189809 +0000 UTC m=+67.763408353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-sjplg" (UID: "cad8511f-d5e6-46ee-820f-e699df814e7f") : secret "samples-operator-tls" not found
Sep 16 00:34:41.702475 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.702431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:41.702653 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.702525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:41.702653 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.702616    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:41.702798 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.702682    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 00:34:41.702798 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.702702    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls podName:cc191bfb-a7ab-4ce4-994b-518fa5b494df nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.702680495 +0000 UTC m=+67.864190749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fkmdm" (UID: "cc191bfb-a7ab-4ce4-994b-518fa5b494df") : secret "cluster-monitoring-operator-tls" not found
Sep 16 00:34:41.702798 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.702760    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls podName:b85b14f1-369a-4c29-9c33-91420ee78f85 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.702741802 +0000 UTC m=+67.864252070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls") pod "dns-default-5qctk" (UID: "b85b14f1-369a-4c29-9c33-91420ee78f85") : secret "dns-default-metrics-tls" not found
Sep 16 00:34:41.803927 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.803892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:41.804147 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.803977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:41.804147 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:41.804018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:41.804147 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.804058    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 00:34:41.804147 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.804110    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 00:34:41.804147 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.804136    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert podName:665ca7f3-d81e-4286-b5ce-1074c872f927 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.804114684 +0000 UTC m=+67.965624938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mb5h7" (UID: "665ca7f3-d81e-4286-b5ce-1074c872f927") : secret "networking-console-plugin-cert" not found
Sep 16 00:34:41.804408 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.804155    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.804142142 +0000 UTC m=+67.965652392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : secret "router-metrics-certs-default" not found
Sep 16 00:34:41.804408 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:41.804168    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle podName:864ca34e-436f-4a8d-a0b0-d3349b46a4ef nodeName:}" failed. No retries permitted until 2026-09-16 00:34:57.80416232 +0000 UTC m=+67.965672570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle") pod "router-default-5475d44bcb-9c4pt" (UID: "864ca34e-436f-4a8d-a0b0-d3349b46a4ef") : configmap references non-existent config key: service-ca.crt
Sep 16 00:34:44.430880 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:44.430813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:44.431435 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:44.430993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:44.431435 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:44.431159    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:44.431435 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:34:44.431247    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls podName:a0227a00-e4a5-433a-a5b7-c7020fdd10f5 nodeName:}" failed. No retries permitted until 2026-09-16 00:34:52.431225572 +0000 UTC m=+62.592735842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hdpd7" (UID: "a0227a00-e4a5-433a-a5b7-c7020fdd10f5") : secret "insights-runtime-extractor-tls" not found
Sep 16 00:34:44.431557 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:44.431477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:51.601140 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:51.601109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvkcq"
Sep 16 00:34:52.328264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.328229    2577 scope.go:117] "RemoveContainer" containerID="2dffcf7c357132d8401783179665fb8e80132f0c71c312d5cc5755d8797f9b4e"
Sep 16 00:34:52.505369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.505326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:52.507843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.507822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0227a00-e4a5-433a-a5b7-c7020fdd10f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hdpd7\" (UID: \"a0227a00-e4a5-433a-a5b7-c7020fdd10f5\") " pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:52.570310 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.570271    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4hl7\""
Sep 16 00:34:52.578469 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.578409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hdpd7"
Sep 16 00:34:52.765712 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.765686    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:34:52.766103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.765742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" event={"ID":"4305e130-063f-4318-a07c-7a6cb748c1ac","Type":"ContainerStarted","Data":"b4e05076fc0e2ac9ac91e87024f34fa3022ebde9969813c7fc8195aa0042b84a"}
Sep 16 00:34:52.766103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.766008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:52.783025 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.782976    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-hzw72" podStartSLOduration=33.994645893 podStartE2EDuration="41.782943182s" totalImagesPullingTime="7.788297289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:52.781576485 +0000 UTC m=+62.943086757" watchObservedRunningTime="2026-09-16 00:34:52.782943182 +0000 UTC m=+62.944453451"
Sep 16 00:34:52.849368 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:52.849319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hdpd7"]
Sep 16 00:34:52.851847 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:52.851821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0227a00_e4a5_433a_a5b7_c7020fdd10f5.slice/crio-ad7d3dd15b165b84531e15a039e2ff9400364cdf8d3a8a086534e9a987b39a3a WatchSource:0}: Error finding container ad7d3dd15b165b84531e15a039e2ff9400364cdf8d3a8a086534e9a987b39a3a: Status 404 returned error can't find the container with id ad7d3dd15b165b84531e15a039e2ff9400364cdf8d3a8a086534e9a987b39a3a
Sep 16 00:34:53.419591 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:53.419559    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-hzw72"
Sep 16 00:34:53.771139 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:53.771101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdpd7" event={"ID":"a0227a00-e4a5-433a-a5b7-c7020fdd10f5","Type":"ContainerStarted","Data":"b746f7631b067a88ca1f492b83ab4edfab39d780d058b7d9572e915203ca5276"}
Sep 16 00:34:53.771549 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:53.771184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdpd7" event={"ID":"a0227a00-e4a5-433a-a5b7-c7020fdd10f5","Type":"ContainerStarted","Data":"6c4d78149f35ffc0eed6a483f6d6b966e7edca03d9970d5e0acc5d2c2c5819f9"}
Sep 16 00:34:53.771549 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:53.771205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdpd7" event={"ID":"a0227a00-e4a5-433a-a5b7-c7020fdd10f5","Type":"ContainerStarted","Data":"ad7d3dd15b165b84531e15a039e2ff9400364cdf8d3a8a086534e9a987b39a3a"}
Sep 16 00:34:55.032607 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.032565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:55.032881 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.032641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:55.035455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.035436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 00:34:55.036222 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.036202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 00:34:55.045453 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.045425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6539572a-35ca-4938-a691-21ff110ed047-original-pull-secret\") pod \"global-pull-secret-syncer-z2xsn\" (UID: \"6539572a-35ca-4938-a691-21ff110ed047\") " pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:55.045557 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.045461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12-metrics-certs\") pod \"network-metrics-daemon-p9nw7\" (UID: \"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12\") " pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:55.133798 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.133758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:55.136341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.136318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgzt\" (UniqueName: \"kubernetes.io/projected/1f59ef6a-348f-4b5b-9831-e8f3038a4d1e-kube-api-access-gfgzt\") pod \"network-check-target-ftzw9\" (UID: \"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e\") " pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:55.248508 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.248466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2xsn"
Sep 16 00:34:55.261203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.261179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7jvk\""
Sep 16 00:34:55.269170 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.269147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9nw7"
Sep 16 00:34:55.269572 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.269548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wgg7z\""
Sep 16 00:34:55.277610 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.277581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:55.526244 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.526073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2xsn"]
Sep 16 00:34:55.528769 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:55.528744    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6539572a_35ca_4938_a691_21ff110ed047.slice/crio-37e231419faaab4e9a739d331850648fa8d8fc34db32fb30eefbdac4b89f7216 WatchSource:0}: Error finding container 37e231419faaab4e9a739d331850648fa8d8fc34db32fb30eefbdac4b89f7216: Status 404 returned error can't find the container with id 37e231419faaab4e9a739d331850648fa8d8fc34db32fb30eefbdac4b89f7216
Sep 16 00:34:55.556777 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.556653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ftzw9"]
Sep 16 00:34:55.558128 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.558105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9nw7"]
Sep 16 00:34:55.559233 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:55.559205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5368dd_4c5f_4594_bbaf_d7a3c5cefb12.slice/crio-30386150a6c00c8f46a0644524961752b27c752b33a7742af94c3c0f8be36fe2 WatchSource:0}: Error finding container 30386150a6c00c8f46a0644524961752b27c752b33a7742af94c3c0f8be36fe2: Status 404 returned error can't find the container with id 30386150a6c00c8f46a0644524961752b27c752b33a7742af94c3c0f8be36fe2
Sep 16 00:34:55.559867 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:55.559846    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f59ef6a_348f_4b5b_9831_e8f3038a4d1e.slice/crio-c8beae6a3bb6c6dbdf89d7c81d0248212492a13c8bbde74a4714718ef3b92932 WatchSource:0}: Error finding container c8beae6a3bb6c6dbdf89d7c81d0248212492a13c8bbde74a4714718ef3b92932: Status 404 returned error can't find the container with id c8beae6a3bb6c6dbdf89d7c81d0248212492a13c8bbde74a4714718ef3b92932
Sep 16 00:34:55.779845 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.779810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2xsn" event={"ID":"6539572a-35ca-4938-a691-21ff110ed047","Type":"ContainerStarted","Data":"37e231419faaab4e9a739d331850648fa8d8fc34db32fb30eefbdac4b89f7216"}
Sep 16 00:34:55.780961 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.780916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9nw7" event={"ID":"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12","Type":"ContainerStarted","Data":"30386150a6c00c8f46a0644524961752b27c752b33a7742af94c3c0f8be36fe2"}
Sep 16 00:34:55.783046 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.783014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hdpd7" event={"ID":"a0227a00-e4a5-433a-a5b7-c7020fdd10f5","Type":"ContainerStarted","Data":"b975fc7d683d1bd6b00d11352c94408d33d7b6b90456ba39477da2fa21a2ef9d"}
Sep 16 00:34:55.784434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.784413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ftzw9" event={"ID":"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e","Type":"ContainerStarted","Data":"40f6c4d89059c402ed23e3d5998a72bcd2cac5035d3f805a18e48b918e7f03a4"}
Sep 16 00:34:55.784520 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.784449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ftzw9" event={"ID":"1f59ef6a-348f-4b5b-9831-e8f3038a4d1e","Type":"ContainerStarted","Data":"c8beae6a3bb6c6dbdf89d7c81d0248212492a13c8bbde74a4714718ef3b92932"}
Sep 16 00:34:55.784570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.784539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:34:55.801079 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.801030    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hdpd7" podStartSLOduration=17.733803695 podStartE2EDuration="19.801015126s" totalImagesPullingTime="2.067211431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:34:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:55.799764488 +0000 UTC m=+65.961274760" watchObservedRunningTime="2026-09-16 00:34:55.801015126 +0000 UTC m=+65.962525398"
Sep 16 00:34:55.813145 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:55.813052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ftzw9" podStartSLOduration=64.813037034 podStartE2EDuration="1m4.813037034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:55.812600104 +0000 UTC m=+65.974110375" watchObservedRunningTime="2026-09-16 00:34:55.813037034 +0000 UTC m=+65.974547359"
Sep 16 00:34:57.583601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.583553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"]
Sep 16 00:34:57.586120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.586095    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"]
Sep 16 00:34:57.586259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.586237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.589823 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.589588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.589823 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.589683    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 00:34:57.589823 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.589685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 00:34:57.591351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.591033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-v2np8\""
Sep 16 00:34:57.591351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.591092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 00:34:57.591351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.591218    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 00:34:57.591800 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.591779    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"]
Sep 16 00:34:57.593973 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.593930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 00:34:57.594445 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.594425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.597358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.596980    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 00:34:57.597358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.597127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 00:34:57.597358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.597196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 00:34:57.597358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.597284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 00:34:57.600131 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.600106    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"]
Sep 16 00:34:57.603169 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.603146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"]
Sep 16 00:34:57.620766 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.620735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"]
Sep 16 00:34:57.660390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.660338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:57.660590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.660500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:57.663638 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.663606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"image-registry-5948fb99cf-nl9jx\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") " pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:57.663781 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.663672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cad8511f-d5e6-46ee-820f-e699df814e7f-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-sjplg\" (UID: \"cad8511f-d5e6-46ee-820f-e699df814e7f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:57.686898 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.686859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-ldwk9"]
Sep 16 00:34:57.689629 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.689561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:34:57.692141 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.692115    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 00:34:57.692277 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.692221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-97llp\""
Sep 16 00:34:57.692350 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.692329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 00:34:57.701052 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.701024    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ldwk9"]
Sep 16 00:34:57.762006 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.761938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3745c09e-c888-4c43-b36c-d6c5cd573b96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:57.762192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e5e3cf5-16b3-484f-8724-ba130622fd1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.762192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947ac073-bdee-4ad6-9155-d2a16da604dc-tmp\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.762192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49vz\" (UniqueName: \"kubernetes.io/projected/7e5e3cf5-16b3-484f-8724-ba130622fd1b-kube-api-access-m49vz\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.762382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgqd\" (UniqueName: \"kubernetes.io/projected/3745c09e-c888-4c43-b36c-d6c5cd573b96-kube-api-access-ncgqd\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:57.762592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.762592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/947ac073-bdee-4ad6-9155-d2a16da604dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.762592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.762519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xmc\" (UniqueName: \"kubernetes.io/projected/947ac073-bdee-4ad6-9155-d2a16da604dc-kube-api-access-f6xmc\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.764910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.764889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b85b14f1-369a-4c29-9c33-91420ee78f85-metrics-tls\") pod \"dns-default-5qctk\" (UID: \"b85b14f1-369a-4c29-9c33-91420ee78f85\") " pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:57.765134 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.765112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc191bfb-a7ab-4ce4-994b-518fa5b494df-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fkmdm\" (UID: \"cc191bfb-a7ab-4ce4-994b-518fa5b494df\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:57.795145 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.795101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9nw7" event={"ID":"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12","Type":"ContainerStarted","Data":"0b626ada7af783b10af7b66b110b49de82ca87f0b06fab55e7244af3a1f22000"}
Sep 16 00:34:57.795296 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.795171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9nw7" event={"ID":"1d5368dd-4c5f-4594-bbaf-d7a3c5cefb12","Type":"ContainerStarted","Data":"e57596d2d421c270f2ccacfac2e9dff564baf6826a8314430543bd608ca29e3f"}
Sep 16 00:34:57.812257 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.812193    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p9nw7" podStartSLOduration=65.188604149 podStartE2EDuration="1m6.812169278s" totalImagesPullingTime="1.623565129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:57.811682884 +0000 UTC m=+67.973193158" watchObservedRunningTime="2026-09-16 00:34:57.812169278 +0000 UTC m=+67.973679553"
Sep 16 00:34:57.828139 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.828108    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sptnt\""
Sep 16 00:34:57.836120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.836043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:57.863420 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49vz\" (UniqueName: \"kubernetes.io/projected/7e5e3cf5-16b3-484f-8724-ba130622fd1b-kube-api-access-m49vz\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.863593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:57.863593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.863593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.863593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgqd\" (UniqueName: \"kubernetes.io/projected/3745c09e-c888-4c43-b36c-d6c5cd573b96-kube-api-access-ncgqd\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.863593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.863851 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.863851 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/947ac073-bdee-4ad6-9155-d2a16da604dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.863851 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.863677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpcz\" (UniqueName: \"kubernetes.io/projected/412a03f2-398c-4be0-a7f6-0765c7da8310-kube-api-access-fxpcz\") pod \"downloads-76f99584f4-ldwk9\" (UID: \"412a03f2-398c-4be0-a7f6-0765c7da8310\") " pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:34:57.864335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.864291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-service-ca-bundle\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:57.867673 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.867633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.867809 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.867783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.867886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.867864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/947ac073-bdee-4ad6-9155-d2a16da604dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.868149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:57.868244 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xmc\" (UniqueName: \"kubernetes.io/projected/947ac073-bdee-4ad6-9155-d2a16da604dc-kube-api-access-f6xmc\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.868244 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3745c09e-c888-4c43-b36c-d6c5cd573b96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.868349 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:57.868349 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e5e3cf5-16b3-484f-8724-ba130622fd1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.868349 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947ac073-bdee-4ad6-9155-d2a16da604dc-tmp\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.868591 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/947ac073-bdee-4ad6-9155-d2a16da604dc-tmp\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.868815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.868791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-hub\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.869495 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.869471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3745c09e-c888-4c43-b36c-d6c5cd573b96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.871543 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.871516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/665ca7f3-d81e-4286-b5ce-1074c872f927-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mb5h7\" (UID: \"665ca7f3-d81e-4286-b5ce-1074c872f927\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:57.871920 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.871889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864ca34e-436f-4a8d-a0b0-d3349b46a4ef-metrics-certs\") pod \"router-default-5475d44bcb-9c4pt\" (UID: \"864ca34e-436f-4a8d-a0b0-d3349b46a4ef\") " pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:57.872654 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.872631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3745c09e-c888-4c43-b36c-d6c5cd573b96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.873370 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.873346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e5e3cf5-16b3-484f-8724-ba130622fd1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.876021 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.875999    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-whctm\""
Sep 16 00:34:57.879629 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.879589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgqd\" (UniqueName: \"kubernetes.io/projected/3745c09e-c888-4c43-b36c-d6c5cd573b96-kube-api-access-ncgqd\") pod \"cluster-proxy-proxy-agent-64847f56b9-j2r57\" (UID: \"3745c09e-c888-4c43-b36c-d6c5cd573b96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.880353 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.880308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xmc\" (UniqueName: \"kubernetes.io/projected/947ac073-bdee-4ad6-9155-d2a16da604dc-kube-api-access-f6xmc\") pod \"klusterlet-addon-workmgr-7c854cf874-njn2j\" (UID: \"947ac073-bdee-4ad6-9155-d2a16da604dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.880731 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.880711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49vz\" (UniqueName: \"kubernetes.io/projected/7e5e3cf5-16b3-484f-8724-ba130622fd1b-kube-api-access-m49vz\") pod \"managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q\" (UID: \"7e5e3cf5-16b3-484f-8724-ba130622fd1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.884488 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.884471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"
Sep 16 00:34:57.909861 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.909720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"
Sep 16 00:34:57.920149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.919995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:34:57.927337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.927144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"
Sep 16 00:34:57.945741 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.944786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l9n8m\""
Sep 16 00:34:57.953016 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.952764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"
Sep 16 00:34:57.971871 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.969340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpcz\" (UniqueName: \"kubernetes.io/projected/412a03f2-398c-4be0-a7f6-0765c7da8310-kube-api-access-fxpcz\") pod \"downloads-76f99584f4-ldwk9\" (UID: \"412a03f2-398c-4be0-a7f6-0765c7da8310\") " pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:34:57.986746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.986480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rpgmh\""
Sep 16 00:34:57.993210 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.992993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qctk"
Sep 16 00:34:57.998677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:57.998451    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kmpkg\""
Sep 16 00:34:58.001883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.001853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpcz\" (UniqueName: \"kubernetes.io/projected/412a03f2-398c-4be0-a7f6-0765c7da8310-kube-api-access-fxpcz\") pod \"downloads-76f99584f4-ldwk9\" (UID: \"412a03f2-398c-4be0-a7f6-0765c7da8310\") " pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:34:58.006356 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.006330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"
Sep 16 00:34:58.020714 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.019148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:34:58.022040 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.021785    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z9xb4\""
Sep 16 00:34:58.031206 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.031107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:58.126218 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.126075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:34:58.145714 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.145119    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf050e9_b781_4881_b040_104f2fa91926.slice/crio-30cbbfb146b5e5aa45416ca8e1bffc704ff6ad62e9b4ddab107f071edb41e8e4 WatchSource:0}: Error finding container 30cbbfb146b5e5aa45416ca8e1bffc704ff6ad62e9b4ddab107f071edb41e8e4: Status 404 returned error can't find the container with id 30cbbfb146b5e5aa45416ca8e1bffc704ff6ad62e9b4ddab107f071edb41e8e4
Sep 16 00:34:58.189282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.186479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg"]
Sep 16 00:34:58.225843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.225814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q"]
Sep 16 00:34:58.229107 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.227668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"]
Sep 16 00:34:58.233361 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.231247    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e5e3cf5_16b3_484f_8724_ba130622fd1b.slice/crio-9a8315c45b54cb698d3d93209f639f25e2d4624f5e8d461fb8ea8320de29bc3d WatchSource:0}: Error finding container 9a8315c45b54cb698d3d93209f639f25e2d4624f5e8d461fb8ea8320de29bc3d: Status 404 returned error can't find the container with id 9a8315c45b54cb698d3d93209f639f25e2d4624f5e8d461fb8ea8320de29bc3d
Sep 16 00:34:58.233361 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.232906    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947ac073_bdee_4ad6_9155_d2a16da604dc.slice/crio-c37ed2df62fdb253ba30e7cc52add907eeb411dea505f09ee7a9d9c55d8a11ac WatchSource:0}: Error finding container c37ed2df62fdb253ba30e7cc52add907eeb411dea505f09ee7a9d9c55d8a11ac: Status 404 returned error can't find the container with id c37ed2df62fdb253ba30e7cc52add907eeb411dea505f09ee7a9d9c55d8a11ac
Sep 16 00:34:58.265129 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.264833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57"]
Sep 16 00:34:58.265510 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.265479    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3745c09e_c888_4c43_b36c_d6c5cd573b96.slice/crio-e6eb7deb2f8004839cb5c1efdde95614340d4ab46b8f7aee75c53e2e480904f8 WatchSource:0}: Error finding container e6eb7deb2f8004839cb5c1efdde95614340d4ab46b8f7aee75c53e2e480904f8: Status 404 returned error can't find the container with id e6eb7deb2f8004839cb5c1efdde95614340d4ab46b8f7aee75c53e2e480904f8
Sep 16 00:34:58.279820 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.279790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm"]
Sep 16 00:34:58.325094 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.325061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qctk"]
Sep 16 00:34:58.329130 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.329101    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85b14f1_369a_4c29_9c33_91420ee78f85.slice/crio-06c935b8de3ec7d66bee3855e84ec0f3936d64d6edca22f1ede75067681375e6 WatchSource:0}: Error finding container 06c935b8de3ec7d66bee3855e84ec0f3936d64d6edca22f1ede75067681375e6: Status 404 returned error can't find the container with id 06c935b8de3ec7d66bee3855e84ec0f3936d64d6edca22f1ede75067681375e6
Sep 16 00:34:58.343406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.343372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7"]
Sep 16 00:34:58.346591 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.346554    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665ca7f3_d81e_4286_b5ce_1074c872f927.slice/crio-56e2e0606cd15150aa9d35772ee4196b7594804db621adb0d921240787826e18 WatchSource:0}: Error finding container 56e2e0606cd15150aa9d35772ee4196b7594804db621adb0d921240787826e18: Status 404 returned error can't find the container with id 56e2e0606cd15150aa9d35772ee4196b7594804db621adb0d921240787826e18
Sep 16 00:34:58.383406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.383375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ldwk9"]
Sep 16 00:34:58.384719 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.384696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5475d44bcb-9c4pt"]
Sep 16 00:34:58.386646 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:34:58.386619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412a03f2_398c_4be0_a7f6_0765c7da8310.slice/crio-d4e9d4adeac0973432ec0471a21fb7d9ee2a4d9512c1ab1431964701a49ffd18 WatchSource:0}: Error finding container d4e9d4adeac0973432ec0471a21fb7d9ee2a4d9512c1ab1431964701a49ffd18: Status 404 returned error can't find the container with id d4e9d4adeac0973432ec0471a21fb7d9ee2a4d9512c1ab1431964701a49ffd18
Sep 16 00:34:58.802394 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.802285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm" event={"ID":"cc191bfb-a7ab-4ce4-994b-518fa5b494df","Type":"ContainerStarted","Data":"f1bc161aea2bef3d3de75d492cc263a6f21e3879581fbccd6a845fabb5f61fc0"}
Sep 16 00:34:58.804685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.804646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57" event={"ID":"3745c09e-c888-4c43-b36c-d6c5cd573b96","Type":"ContainerStarted","Data":"e6eb7deb2f8004839cb5c1efdde95614340d4ab46b8f7aee75c53e2e480904f8"}
Sep 16 00:34:58.806375 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.806345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q" event={"ID":"7e5e3cf5-16b3-484f-8724-ba130622fd1b","Type":"ContainerStarted","Data":"9a8315c45b54cb698d3d93209f639f25e2d4624f5e8d461fb8ea8320de29bc3d"}
Sep 16 00:34:58.807999 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.807964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg" event={"ID":"cad8511f-d5e6-46ee-820f-e699df814e7f","Type":"ContainerStarted","Data":"a2ab9fb519384bbb671a27e3d2be43480817767774d3678000603296c00332b4"}
Sep 16 00:34:58.810184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.810152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5475d44bcb-9c4pt" event={"ID":"864ca34e-436f-4a8d-a0b0-d3349b46a4ef","Type":"ContainerStarted","Data":"8cea5e10556ea1adf0a0da85ee621003d66e0a4380e1fafaac78e2dd7024c7a1"}
Sep 16 00:34:58.810300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.810206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5475d44bcb-9c4pt" event={"ID":"864ca34e-436f-4a8d-a0b0-d3349b46a4ef","Type":"ContainerStarted","Data":"65e8aded19786113cf966e51b25ab9c8f7a3dfd993e66a43f978887d7df0008b"}
Sep 16 00:34:58.812944 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.812861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j" event={"ID":"947ac073-bdee-4ad6-9155-d2a16da604dc","Type":"ContainerStarted","Data":"c37ed2df62fdb253ba30e7cc52add907eeb411dea505f09ee7a9d9c55d8a11ac"}
Sep 16 00:34:58.815140 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.815112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" event={"ID":"4cf050e9-b781-4881-b040-104f2fa91926","Type":"ContainerStarted","Data":"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"}
Sep 16 00:34:58.815218 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.815153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" event={"ID":"4cf050e9-b781-4881-b040-104f2fa91926","Type":"ContainerStarted","Data":"30cbbfb146b5e5aa45416ca8e1bffc704ff6ad62e9b4ddab107f071edb41e8e4"}
Sep 16 00:34:58.815343 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.815321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:34:58.817017 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.816965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7" event={"ID":"665ca7f3-d81e-4286-b5ce-1074c872f927","Type":"ContainerStarted","Data":"56e2e0606cd15150aa9d35772ee4196b7594804db621adb0d921240787826e18"}
Sep 16 00:34:58.818844 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.818821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qctk" event={"ID":"b85b14f1-369a-4c29-9c33-91420ee78f85","Type":"ContainerStarted","Data":"06c935b8de3ec7d66bee3855e84ec0f3936d64d6edca22f1ede75067681375e6"}
Sep 16 00:34:58.821110 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.821050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ldwk9" event={"ID":"412a03f2-398c-4be0-a7f6-0765c7da8310","Type":"ContainerStarted","Data":"d4e9d4adeac0973432ec0471a21fb7d9ee2a4d9512c1ab1431964701a49ffd18"}
Sep 16 00:34:58.847945 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.847875    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" podStartSLOduration=67.847850083 podStartE2EDuration="1m7.847850083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:58.846096569 +0000 UTC m=+69.007606853" watchObservedRunningTime="2026-09-16 00:34:58.847850083 +0000 UTC m=+69.009360358"
Sep 16 00:34:58.848202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:58.848172    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5475d44bcb-9c4pt" podStartSLOduration=47.848163572 podStartE2EDuration="47.848163572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:34:58.828616263 +0000 UTC m=+68.990126536" watchObservedRunningTime="2026-09-16 00:34:58.848163572 +0000 UTC m=+69.009673867"
Sep 16 00:34:59.032337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:59.032296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:59.035197 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:59.035165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:59.825764 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:59.825703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:34:59.827031 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:34:59.827007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5475d44bcb-9c4pt"
Sep 16 00:35:09.901145 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.900320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7" event={"ID":"665ca7f3-d81e-4286-b5ce-1074c872f927","Type":"ContainerStarted","Data":"56de53527a4449131fe55288424446dc8209692af281b6381afa141620b67339"}
Sep 16 00:35:09.906681 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.906601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qctk" event={"ID":"b85b14f1-369a-4c29-9c33-91420ee78f85","Type":"ContainerStarted","Data":"475f8fd48ea2d125bf0644356c3e8e1361f03bc44d1ebcc9f4d87920fdaac00b"}
Sep 16 00:35:09.909996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.909512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2xsn" event={"ID":"6539572a-35ca-4938-a691-21ff110ed047","Type":"ContainerStarted","Data":"d07acca7fd4b8e1638d950b042572aca261fb981d1db01c4e102229f979aa725"}
Sep 16 00:35:09.915012 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.914989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm" event={"ID":"cc191bfb-a7ab-4ce4-994b-518fa5b494df","Type":"ContainerStarted","Data":"544d32f9afcbeef20b22c8e9b214d6811d6a8217492886d786ccb43bda6eb09f"}
Sep 16 00:35:09.919987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.919081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mb5h7" podStartSLOduration=48.28394272 podStartE2EDuration="58.919038086s" totalImagesPullingTime="10.635095366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:09.918997764 +0000 UTC m=+80.080508037" watchObservedRunningTime="2026-09-16 00:35:09.919038086 +0000 UTC m=+80.080548360"
Sep 16 00:35:09.919987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.919524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57" event={"ID":"3745c09e-c888-4c43-b36c-d6c5cd573b96","Type":"ContainerStarted","Data":"022afe78c9aecfb838dd08ada1e3931b2134701a8aac917ac3b31adb35c4ec5a"}
Sep 16 00:35:09.919987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.919563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57" event={"ID":"3745c09e-c888-4c43-b36c-d6c5cd573b96","Type":"ContainerStarted","Data":"1157118bb22e0ea44c12fb6341921812b5431ac9887ab79725d89bdfc567d2f3"}
Sep 16 00:35:09.922154 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.921590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q" event={"ID":"7e5e3cf5-16b3-484f-8724-ba130622fd1b","Type":"ContainerStarted","Data":"a9e34cc5be184e0f63a02f0b62cb3d3976536f4b57a5782ebdea4c66baea86e2"}
Sep 16 00:35:09.927391 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.926272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg" event={"ID":"cad8511f-d5e6-46ee-820f-e699df814e7f","Type":"ContainerStarted","Data":"09b69cc02300d4880b1e4fab4867fb27bf82a0655db2a73b4b4b8550cf5f9770"}
Sep 16 00:35:09.927391 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.926327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg" event={"ID":"cad8511f-d5e6-46ee-820f-e699df814e7f","Type":"ContainerStarted","Data":"29e62d88b65bc49d13b51df8994e6859d4f6c4cce6fd2239d8fa930bd17dfad4"}
Sep 16 00:35:09.935672 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.935616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j" event={"ID":"947ac073-bdee-4ad6-9155-d2a16da604dc","Type":"ContainerStarted","Data":"007dbd32686ccf925e368c5bea2340bb140e6c8e9546b384a62bcfd6e44df013"}
Sep 16 00:35:09.936870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.936795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:35:09.939561 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.939515    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j"
Sep 16 00:35:09.958645 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.958585    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fkmdm" podStartSLOduration=54.675898621 podStartE2EDuration="58.958562815s" totalImagesPullingTime="4.282664194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:09.957868087 +0000 UTC m=+80.119378350" watchObservedRunningTime="2026-09-16 00:35:09.958562815 +0000 UTC m=+80.120073088"
Sep 16 00:35:09.974582 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.974490    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z2xsn" podStartSLOduration=71.939698636 podStartE2EDuration="1m18.974466123s" totalImagesPullingTime="7.034767487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 00:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:09.973413264 +0000 UTC m=+80.134923537" watchObservedRunningTime="2026-09-16 00:35:09.974466123 +0000 UTC m=+80.135976397"
Sep 16 00:35:09.991679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:09.991244    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cdd8d-lnk4q" podStartSLOduration=1.721778614 podStartE2EDuration="12.99122159s" totalImagesPullingTime="11.269442976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:09.990479526 +0000 UTC m=+80.151989798" watchObservedRunningTime="2026-09-16 00:35:09.99122159 +0000 UTC m=+80.152731863"
Sep 16 00:35:10.012573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.012509    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-sjplg" podStartSLOduration=54.680959097 podStartE2EDuration="59.012487539s" totalImagesPullingTime="4.331528442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:10.011249193 +0000 UTC m=+80.172759466" watchObservedRunningTime="2026-09-16 00:35:10.012487539 +0000 UTC m=+80.173997845"
Sep 16 00:35:10.030630 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.030574    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c854cf874-njn2j" podStartSLOduration=1.732408999 podStartE2EDuration="13.030559911s" totalImagesPullingTime="11.298150912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:10.029303768 +0000 UTC m=+80.190814040" watchObservedRunningTime="2026-09-16 00:35:10.030559911 +0000 UTC m=+80.192070182"
Sep 16 00:35:10.943762 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.943722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57" event={"ID":"3745c09e-c888-4c43-b36c-d6c5cd573b96","Type":"ContainerStarted","Data":"77bb18cef6a2a9690173a557c475734f65f541f7d3b1ce4c42d38583de312e26"}
Sep 16 00:35:10.946430 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.946393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qctk" event={"ID":"b85b14f1-369a-4c29-9c33-91420ee78f85","Type":"ContainerStarted","Data":"6941b395651fc548fb3a7b247abf2bf94d84a4fdce11ad154ce2e994ef62c5f1"}
Sep 16 00:35:10.947336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.947316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5qctk"
Sep 16 00:35:10.962303 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.962243    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64847f56b9-j2r57" podStartSLOduration=2.7421259190000002 podStartE2EDuration="13.962221955s" totalImagesPullingTime="11.220096036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:10.960911947 +0000 UTC m=+81.122422223" watchObservedRunningTime="2026-09-16 00:35:10.962221955 +0000 UTC m=+81.123732230"
Sep 16 00:35:10.978113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:10.978044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5qctk" podStartSLOduration=34.821737256 podStartE2EDuration="45.978027453s" totalImagesPullingTime="11.156290197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:10.977906917 +0000 UTC m=+81.139417214" watchObservedRunningTime="2026-09-16 00:35:10.978027453 +0000 UTC m=+81.139537726"
Sep 16 00:35:17.841621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:17.841481    2577 patch_prober.go:28] interesting pod/image-registry-5948fb99cf-nl9jx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 00:35:17.841621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:17.841550    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" podUID="4cf050e9-b781-4881-b040-104f2fa91926" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:35:18.620985 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.620931    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gb65z"]
Sep 16 00:35:18.651257 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.651220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.657383 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.657351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f98b9\""
Sep 16 00:35:18.657531 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.657365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 00:35:18.657631 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.657612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 00:35:18.657876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.657851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 00:35:18.659224 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.658558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 00:35:18.758161 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlsdm\" (UniqueName: \"kubernetes.io/projected/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-kube-api-access-xlsdm\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758327 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758411 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758411 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-tls\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758411 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-wtmp\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-textfile\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-sys\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-metrics-client-ca\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.758522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.758510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-root\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-tls\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-wtmp\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-textfile\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-sys\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-metrics-client-ca\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-root\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlsdm\" (UniqueName: \"kubernetes.io/projected/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-kube-api-access-xlsdm\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.859767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.860350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-sys\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.860662 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.860385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-root\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.861611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.860922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-wtmp\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.861611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.860988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-metrics-client-ca\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.861611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.861242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-textfile\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.861611 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.861513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.864525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.864474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.866366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.866313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-node-exporter-tls\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.869389 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.869341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlsdm\" (UniqueName: \"kubernetes.io/projected/73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c-kube-api-access-xlsdm\") pod \"node-exporter-gb65z\" (UID: \"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c\") " pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:18.968867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:18.968781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gb65z"
Sep 16 00:35:19.831156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:19.831122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:35:20.385692 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.385565    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:35:20.995447 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.995396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gb65z" event={"ID":"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c","Type":"ContainerStarted","Data":"53ddd5568b9c149d484758875ab29a03daaad76a6682f7577ceb3d432d074521"}
Sep 16 00:35:20.997055 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.997022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ldwk9" event={"ID":"412a03f2-398c-4be0-a7f6-0765c7da8310","Type":"ContainerStarted","Data":"06f272479817817a5d6fc8c2e3706b410a0d7624da52ffef162066a7906ba7a9"}
Sep 16 00:35:20.997393 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.997374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:35:20.999004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.998980    2577 patch_prober.go:28] interesting pod/downloads-76f99584f4-ldwk9 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body=
Sep 16 00:35:20.999088 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:20.999036    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-ldwk9" podUID="412a03f2-398c-4be0-a7f6-0765c7da8310" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 00:35:21.012679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:21.012622    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-ldwk9" podStartSLOduration=1.621417279 podStartE2EDuration="24.01260349s" totalImagesPullingTime="22.391186211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:21.012273391 +0000 UTC m=+91.173783675" watchObservedRunningTime="2026-09-16 00:35:21.01260349 +0000 UTC m=+91.174113763"
Sep 16 00:35:21.960123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:21.960041    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5qctk"
Sep 16 00:35:22.005835 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:22.004700    2577 generic.go:356] "Generic (PLEG): container finished" podID="73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c" containerID="f894f9d51071d148cb70d6da88acc5d6fa58f85174e30a8716bbf9752fad2931" exitCode=0
Sep 16 00:35:22.006639 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:22.005674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gb65z" event={"ID":"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c","Type":"ContainerDied","Data":"f894f9d51071d148cb70d6da88acc5d6fa58f85174e30a8716bbf9752fad2931"}
Sep 16 00:35:22.019619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:22.018829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-ldwk9"
Sep 16 00:35:23.012705 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:23.012659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gb65z" event={"ID":"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c","Type":"ContainerStarted","Data":"914758ecddf6027a7d6f90579540c1a4732d9c3b6b554f733a333625baaeb24b"}
Sep 16 00:35:23.013224 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:23.012734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gb65z" event={"ID":"73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c","Type":"ContainerStarted","Data":"ac1f7edc2e40df4e3598b551fe3cb7a823a4bba4c664e0f4e141c4e777c00326"}
Sep 16 00:35:23.034667 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:23.034599    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gb65z" podStartSLOduration=3.624773714 podStartE2EDuration="5.034578975s" totalImagesPullingTime="1.306249673s" totalInitContainerRuntime="103.555588ms" isStatefulPod=true podCreationTimestamp="2026-09-16 00:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:35:23.032230996 +0000 UTC m=+93.193741270" watchObservedRunningTime="2026-09-16 00:35:23.034578975 +0000 UTC m=+93.196089247"
Sep 16 00:35:26.791690 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:26.791654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ftzw9"
Sep 16 00:35:45.409184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.409142    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" podUID="4cf050e9-b781-4881-b040-104f2fa91926" containerName="registry" containerID="cri-o://a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a" gracePeriod=30
Sep 16 00:35:45.687614 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.687533    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:35:45.706692 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706660    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.706859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.706859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.706859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.706859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf6cf\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.706859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.707155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.707155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.706915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") pod \"4cf050e9-b781-4881-b040-104f2fa91926\" (UID: \"4cf050e9-b781-4881-b040-104f2fa91926\") "
Sep 16 00:35:45.707254 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.707184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:35:45.707312 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.707253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:35:45.709759 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.709722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:35:45.709894 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.709781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:35:45.709894 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.709840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:35:45.710031 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.709932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf" (OuterVolumeSpecName: "kube-api-access-mf6cf") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "kube-api-access-mf6cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:35:45.710265 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.710226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:35:45.716913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.716840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4cf050e9-b781-4881-b040-104f2fa91926" (UID: "4cf050e9-b781-4881-b040-104f2fa91926"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:35:45.808434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808391    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cf050e9-b781-4881-b040-104f2fa91926-ca-trust-extracted\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808427    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-trusted-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808438    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-image-registry-private-configuration\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808448    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cf050e9-b781-4881-b040-104f2fa91926-registry-certificates\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808458    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf6cf\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-kube-api-access-mf6cf\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808467    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cf050e9-b781-4881-b040-104f2fa91926-installation-pull-secrets\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808476    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-bound-sa-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:45.808677 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:45.808484    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cf050e9-b781-4881-b040-104f2fa91926-registry-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:35:46.123724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.123683    2577 generic.go:356] "Generic (PLEG): container finished" podID="4cf050e9-b781-4881-b040-104f2fa91926" containerID="a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a" exitCode=0
Sep 16 00:35:46.123926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.123733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx"
Sep 16 00:35:46.123926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.123777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" event={"ID":"4cf050e9-b781-4881-b040-104f2fa91926","Type":"ContainerDied","Data":"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"}
Sep 16 00:35:46.123926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.123868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5948fb99cf-nl9jx" event={"ID":"4cf050e9-b781-4881-b040-104f2fa91926","Type":"ContainerDied","Data":"30cbbfb146b5e5aa45416ca8e1bffc704ff6ad62e9b4ddab107f071edb41e8e4"}
Sep 16 00:35:46.123926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.123895    2577 scope.go:117] "RemoveContainer" containerID="a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"
Sep 16 00:35:46.125400 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.125378    2577 generic.go:356] "Generic (PLEG): container finished" podID="7cde447c-7ce6-4fa6-98a2-4853d9fd5179" containerID="8237a79dd4177579aeac442daa04a47b68500919878e31eaea895e27b6fb7240" exitCode=0
Sep 16 00:35:46.125492 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.125427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v" event={"ID":"7cde447c-7ce6-4fa6-98a2-4853d9fd5179","Type":"ContainerDied","Data":"8237a79dd4177579aeac442daa04a47b68500919878e31eaea895e27b6fb7240"}
Sep 16 00:35:46.125757 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.125740    2577 scope.go:117] "RemoveContainer" containerID="8237a79dd4177579aeac442daa04a47b68500919878e31eaea895e27b6fb7240"
Sep 16 00:35:46.133074 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.133009    2577 scope.go:117] "RemoveContainer" containerID="a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"
Sep 16 00:35:46.133439 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:35:46.133412    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a\": container with ID starting with a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a not found: ID does not exist" containerID="a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"
Sep 16 00:35:46.133529 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.133443    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a"} err="failed to get container status \"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a\": rpc error: code = NotFound desc = could not find container \"a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a\": container with ID starting with a200499545dca0875c6b139cdc6cf76b6aecde010e6616d31d04a0c8d1aff09a not found: ID does not exist"
Sep 16 00:35:46.157704 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.157673    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:35:46.163883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.163854    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5948fb99cf-nl9jx"]
Sep 16 00:35:46.333570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:46.333533    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf050e9-b781-4881-b040-104f2fa91926" path="/var/lib/kubelet/pods/4cf050e9-b781-4881-b040-104f2fa91926/volumes"
Sep 16 00:35:47.131861 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:47.131820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5dc8v" event={"ID":"7cde447c-7ce6-4fa6-98a2-4853d9fd5179","Type":"ContainerStarted","Data":"54679eaa155ee4660e122b22cc48d6defa4c5795c633f00ecd5dfceefa1e2fcc"}
Sep 16 00:35:56.169536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:56.169445    2577 generic.go:356] "Generic (PLEG): container finished" podID="a633fe59-ba7f-4981-a1a0-0533327434e8" containerID="2d0f68766e4bb9d89e6ba43bf75a35e55c277bb7edca50607a15b34f01499b1b" exitCode=0
Sep 16 00:35:56.169536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:56.169517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-zxsw2" event={"ID":"a633fe59-ba7f-4981-a1a0-0533327434e8","Type":"ContainerDied","Data":"2d0f68766e4bb9d89e6ba43bf75a35e55c277bb7edca50607a15b34f01499b1b"}
Sep 16 00:35:56.169990 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:56.169860    2577 scope.go:117] "RemoveContainer" containerID="2d0f68766e4bb9d89e6ba43bf75a35e55c277bb7edca50607a15b34f01499b1b"
Sep 16 00:35:57.175029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:35:57.174991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-zxsw2" event={"ID":"a633fe59-ba7f-4981-a1a0-0533327434e8","Type":"ContainerStarted","Data":"efb7076b4666ead86bfab46dc04f3452e5e7c5123af019edd23453360098f27e"}
Sep 16 00:36:05.209365 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:36:05.209331    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d826309-8374-4890-8ef0-bd8e1ba0e247" containerID="69fe8edca3c3adbe6ab53e1331b64b8eae54c01e02d6059a5e992611bdf92acc" exitCode=0
Sep 16 00:36:05.209829 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:36:05.209404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9" event={"ID":"3d826309-8374-4890-8ef0-bd8e1ba0e247","Type":"ContainerDied","Data":"69fe8edca3c3adbe6ab53e1331b64b8eae54c01e02d6059a5e992611bdf92acc"}
Sep 16 00:36:05.214639 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:36:05.210894    2577 scope.go:117] "RemoveContainer" containerID="69fe8edca3c3adbe6ab53e1331b64b8eae54c01e02d6059a5e992611bdf92acc"
Sep 16 00:36:06.215857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:36:06.215820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9kxj9" event={"ID":"3d826309-8374-4890-8ef0-bd8e1ba0e247","Type":"ContainerStarted","Data":"f02f242845357e3e9e81bab62fa13fbcb2dca345d81a2080178c549a422eba4b"}
Sep 16 00:38:48.492868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.492834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"]
Sep 16 00:38:48.493307 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.493155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf050e9-b781-4881-b040-104f2fa91926" containerName="registry"
Sep 16 00:38:48.493307 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.493166    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf050e9-b781-4881-b040-104f2fa91926" containerName="registry"
Sep 16 00:38:48.493307 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.493219    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf050e9-b781-4881-b040-104f2fa91926" containerName="registry"
Sep 16 00:38:48.495420 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.495400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.498142 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.498120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 00:38:48.498254 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.498141    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 00:38:48.498455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.498441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 00:38:48.498575 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.498560    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 00:38:48.498678 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.498663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zp8vf\""
Sep 16 00:38:48.511825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.511804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"]
Sep 16 00:38:48.584106 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.584071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.584266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.584126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.584266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.584223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.584266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.584259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.584384 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.584281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnp8h\" (UniqueName: \"kubernetes.io/projected/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-kube-api-access-bnp8h\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.685658 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.685625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.685658 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.685664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.685937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.685684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnp8h\" (UniqueName: \"kubernetes.io/projected/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-kube-api-access-bnp8h\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.685937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.685723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.685937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.685754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.686239 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.686210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-manifests\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.688319 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.688289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-tmp\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.688449 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.688432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.688524 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.688506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.697110 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.697086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnp8h\" (UniqueName: \"kubernetes.io/projected/b353ecb1-0c72-4ca4-b1dd-065ae24b1836-kube-api-access-bnp8h\") pod \"opendatahub-operator-controller-manager-55d645f7ff-zt62q\" (UID: \"b353ecb1-0c72-4ca4-b1dd-065ae24b1836\") " pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:48.806143 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:48.806102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:49.080913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:49.080887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"]
Sep 16 00:38:49.083416 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:38:49.083391    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb353ecb1_0c72_4ca4_b1dd_065ae24b1836.slice/crio-b5ad3f7fc733e819f3ac4ef7b864ed43bbae237cf50fc58e3a0efbf1b4e0290d WatchSource:0}: Error finding container b5ad3f7fc733e819f3ac4ef7b864ed43bbae237cf50fc58e3a0efbf1b4e0290d: Status 404 returned error can't find the container with id b5ad3f7fc733e819f3ac4ef7b864ed43bbae237cf50fc58e3a0efbf1b4e0290d
Sep 16 00:38:49.864271 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:49.864222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q" event={"ID":"b353ecb1-0c72-4ca4-b1dd-065ae24b1836","Type":"ContainerStarted","Data":"b5ad3f7fc733e819f3ac4ef7b864ed43bbae237cf50fc58e3a0efbf1b4e0290d"}
Sep 16 00:38:50.269551 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:50.269520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:38:50.275027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:50.274912    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:38:50.287214 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:50.287191    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 00:38:52.880627 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:52.880583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q" event={"ID":"b353ecb1-0c72-4ca4-b1dd-065ae24b1836","Type":"ContainerStarted","Data":"46081dd86019325b31507ae2f9566ba984acb1006dfbe790d5d646108d5111ca"}
Sep 16 00:38:53.886707 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:53.886670    2577 generic.go:356] "Generic (PLEG): container finished" podID="b353ecb1-0c72-4ca4-b1dd-065ae24b1836" containerID="46081dd86019325b31507ae2f9566ba984acb1006dfbe790d5d646108d5111ca" exitCode=0
Sep 16 00:38:53.887203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:53.886759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q" event={"ID":"b353ecb1-0c72-4ca4-b1dd-065ae24b1836","Type":"ContainerDied","Data":"46081dd86019325b31507ae2f9566ba984acb1006dfbe790d5d646108d5111ca"}
Sep 16 00:38:53.887454 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:53.887436    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:38:54.893225 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:54.893188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q" event={"ID":"b353ecb1-0c72-4ca4-b1dd-065ae24b1836","Type":"ContainerStarted","Data":"d4767f88c275f682094e66821e0f6f58076c56f9eda0b0822a5734b062a312ae"}
Sep 16 00:38:54.893706 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:54.893274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:38:54.914732 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:38:54.914668    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q" podStartSLOduration=2.466537252 podStartE2EDuration="6.914649936s" totalImagesPullingTime="3.361048705s" totalInitContainerRuntime="1.087063979s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:38:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:38:54.913301749 +0000 UTC m=+305.074812021" watchObservedRunningTime="2026-09-16 00:38:54.914649936 +0000 UTC m=+305.076160212"
Sep 16 00:39:05.899536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:05.899503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d645f7ff-zt62q"
Sep 16 00:39:08.592178 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.592145    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"]
Sep 16 00:39:08.602340 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.602312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.605266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.605240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 00:39:08.606783 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.606759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 00:39:08.606928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.606859    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6djkb\""
Sep 16 00:39:08.607542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.607524    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"]
Sep 16 00:39:08.761506 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.761464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306f1800-c75b-4910-8922-f5aca47f70f4-tls-certs\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.761718 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.761525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306f1800-c75b-4910-8922-f5aca47f70f4-tmp\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.761718 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.761630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7cw\" (UniqueName: \"kubernetes.io/projected/306f1800-c75b-4910-8922-f5aca47f70f4-kube-api-access-9l7cw\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.863193 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.863095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306f1800-c75b-4910-8922-f5aca47f70f4-tls-certs\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.863193 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.863157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306f1800-c75b-4910-8922-f5aca47f70f4-tmp\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.863193 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.863191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7cw\" (UniqueName: \"kubernetes.io/projected/306f1800-c75b-4910-8922-f5aca47f70f4-kube-api-access-9l7cw\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.865632 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.865605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306f1800-c75b-4910-8922-f5aca47f70f4-tmp\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.865853 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.865833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306f1800-c75b-4910-8922-f5aca47f70f4-tls-certs\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.871774 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.871747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7cw\" (UniqueName: \"kubernetes.io/projected/306f1800-c75b-4910-8922-f5aca47f70f4-kube-api-access-9l7cw\") pod \"kube-auth-proxy-6bb4c8f6-l2fdx\" (UID: \"306f1800-c75b-4910-8922-f5aca47f70f4\") " pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:08.913647 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:08.913604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"
Sep 16 00:39:09.195333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.195304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx"]
Sep 16 00:39:09.197734 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:39:09.197690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306f1800_c75b_4910_8922_f5aca47f70f4.slice/crio-a62b28cd8c57acad05e1be21650127b63b2e4ad957a72859fa09f28f7218cf43 WatchSource:0}: Error finding container a62b28cd8c57acad05e1be21650127b63b2e4ad957a72859fa09f28f7218cf43: Status 404 returned error can't find the container with id a62b28cd8c57acad05e1be21650127b63b2e4ad957a72859fa09f28f7218cf43
Sep 16 00:39:09.684993 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.684929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:09.687482 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.687462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.690178 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.690155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 00:39:09.690367 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.690168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vcsn2\""
Sep 16 00:39:09.690431 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.690170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 00:39:09.690478 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.690183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 00:39:09.690525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.690321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 00:39:09.701726 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.701696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:09.871664 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.871664 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fspl\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.871910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.871910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.871910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.872087 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.872087 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.871984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.965808 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.965536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx" event={"ID":"306f1800-c75b-4910-8922-f5aca47f70f4","Type":"ContainerStarted","Data":"a62b28cd8c57acad05e1be21650127b63b2e4ad957a72859fa09f28f7218cf43"}
Sep 16 00:39:09.973261 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fspl\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.973461 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.973442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.974826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.974760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.976982 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.976933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.977328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.977283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.977850 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.977828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.978209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.978171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.987657 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.987624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fspl\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.998071 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.998018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8jwhb\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:09.998768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:09.998745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:10.294941 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:10.294863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:10.298209 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:39:10.298175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a1afb4_e8f1_4896_9d8a_37fe1212cd4f.slice/crio-14eb7e27cca74c25a683a111c651bbde063e6ce9c2085ae68eb876e16ef8982a WatchSource:0}: Error finding container 14eb7e27cca74c25a683a111c651bbde063e6ce9c2085ae68eb876e16ef8982a: Status 404 returned error can't find the container with id 14eb7e27cca74c25a683a111c651bbde063e6ce9c2085ae68eb876e16ef8982a
Sep 16 00:39:10.972430 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:10.972386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" event={"ID":"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f","Type":"ContainerStarted","Data":"14eb7e27cca74c25a683a111c651bbde063e6ce9c2085ae68eb876e16ef8982a"}
Sep 16 00:39:11.986183 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:11.986135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx" event={"ID":"306f1800-c75b-4910-8922-f5aca47f70f4","Type":"ContainerStarted","Data":"bf245dc0f622f9b079eb300fd4d3b9a0168dd5e6fc32cb071a88d53e6c25aeb5"}
Sep 16 00:39:12.004679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:12.004623    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bb4c8f6-l2fdx" podStartSLOduration=1.3867704 podStartE2EDuration="4.004605999s" totalImagesPullingTime="2.617835599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:12.002370747 +0000 UTC m=+322.163881013" watchObservedRunningTime="2026-09-16 00:39:12.004605999 +0000 UTC m=+322.166116298"
Sep 16 00:39:14.044921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:14.044885    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:39:14.045199 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:14.044978    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:39:15.001366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:15.001327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" event={"ID":"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f","Type":"ContainerStarted","Data":"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"}
Sep 16 00:39:15.001579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:15.001508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:15.003238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:15.003212    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8jwhb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 00:39:15.003342 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:15.003258    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:39:15.045679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:15.045621    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" podStartSLOduration=2.301900857 podStartE2EDuration="6.045604026s" totalImagesPullingTime="3.743703169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:15.044117738 +0000 UTC m=+325.205628029" watchObservedRunningTime="2026-09-16 00:39:15.045604026 +0000 UTC m=+325.207114296"
Sep 16 00:39:16.008099 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:16.008068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:29.545333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.545299    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"]
Sep 16 00:39:29.549043 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.548929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.551500 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.551477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 00:39:29.551637 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.551508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z89qt\""
Sep 16 00:39:29.558472 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.558444    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"]
Sep 16 00:39:29.561777 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.561743    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"]
Sep 16 00:39:29.561941 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.561858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.565134 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.565112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 00:39:29.565335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.565312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-p4rxg\""
Sep 16 00:39:29.581331 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.581243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"]
Sep 16 00:39:29.635050 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.634936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9hg\" (UniqueName: \"kubernetes.io/projected/ea02c116-d193-437b-9841-e97747a744e6-kube-api-access-kn9hg\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.635050 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.635015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea02c116-d193-437b-9841-e97747a744e6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.635057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.635087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6qq\" (UniqueName: \"kubernetes.io/projected/4946dd9c-c8b5-4e77-a870-12138bdb5102-kube-api-access-qm6qq\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.736196 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.736154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea02c116-d193-437b-9841-e97747a744e6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.736408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.736222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.736408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.736261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6qq\" (UniqueName: \"kubernetes.io/projected/4946dd9c-c8b5-4e77-a870-12138bdb5102-kube-api-access-qm6qq\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.736408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.736312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9hg\" (UniqueName: \"kubernetes.io/projected/ea02c116-d193-437b-9841-e97747a744e6-kube-api-access-kn9hg\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.736408 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:39:29.736322    2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 00:39:29.736408 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:39:29.736384    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert podName:4946dd9c-c8b5-4e77-a870-12138bdb5102 nodeName:}" failed. No retries permitted until 2026-09-16 00:39:30.236367668 +0000 UTC m=+340.397877922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert") pod "kserve-controller-manager-d6f4b65b8-r7gtb" (UID: "4946dd9c-c8b5-4e77-a870-12138bdb5102") : secret "kserve-webhook-server-cert" not found
Sep 16 00:39:29.739031 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.738998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea02c116-d193-437b-9841-e97747a744e6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.752333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.752308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9hg\" (UniqueName: \"kubernetes.io/projected/ea02c116-d193-437b-9841-e97747a744e6-kube-api-access-kn9hg\") pod \"llmisvc-controller-manager-777c6bd7d5-v6wsp\" (UID: \"ea02c116-d193-437b-9841-e97747a744e6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:29.753845 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.753823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6qq\" (UniqueName: \"kubernetes.io/projected/4946dd9c-c8b5-4e77-a870-12138bdb5102-kube-api-access-qm6qq\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:29.875406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:29.875372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:30.149616 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:30.149587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"]
Sep 16 00:39:30.151700 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:39:30.151677    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea02c116_d193_437b_9841_e97747a744e6.slice/crio-8b36b5d6ef7829244ba141cc67e7ae8325ef0175478e4d26fb7234fe0d44b195 WatchSource:0}: Error finding container 8b36b5d6ef7829244ba141cc67e7ae8325ef0175478e4d26fb7234fe0d44b195: Status 404 returned error can't find the container with id 8b36b5d6ef7829244ba141cc67e7ae8325ef0175478e4d26fb7234fe0d44b195
Sep 16 00:39:30.239972 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:30.239853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:30.242418 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:30.242392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4946dd9c-c8b5-4e77-a870-12138bdb5102-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7gtb\" (UID: \"4946dd9c-c8b5-4e77-a870-12138bdb5102\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:30.460975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:30.460853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:30.746408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:30.746380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"]
Sep 16 00:39:30.748895 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:39:30.748858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4946dd9c_c8b5_4e77_a870_12138bdb5102.slice/crio-958119ec129f4878aa38f548712d9f6bd66a7ac63c294bd537865f8470b20e19 WatchSource:0}: Error finding container 958119ec129f4878aa38f548712d9f6bd66a7ac63c294bd537865f8470b20e19: Status 404 returned error can't find the container with id 958119ec129f4878aa38f548712d9f6bd66a7ac63c294bd537865f8470b20e19
Sep 16 00:39:31.078613 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.078566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp" event={"ID":"ea02c116-d193-437b-9841-e97747a744e6","Type":"ContainerStarted","Data":"8b36b5d6ef7829244ba141cc67e7ae8325ef0175478e4d26fb7234fe0d44b195"}
Sep 16 00:39:31.079801 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.079773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb" event={"ID":"4946dd9c-c8b5-4e77-a870-12138bdb5102","Type":"ContainerStarted","Data":"958119ec129f4878aa38f548712d9f6bd66a7ac63c294bd537865f8470b20e19"}
Sep 16 00:39:31.543071 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.543031    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:31.543338 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.543285    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerName="discovery" containerID="cri-o://c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3" gracePeriod=30
Sep 16 00:39:31.805927 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.805860    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:31.855419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855665 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fspl\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855665 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855861 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts\") pod \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\" (UID: \"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f\") "
Sep 16 00:39:31.855861 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.855837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:39:31.856130 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.856105    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-ca-configmap\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.858191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858163    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:39:31.858295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts" (OuterVolumeSpecName: "cacerts") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:39:31.858380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs" (OuterVolumeSpecName: "local-certs") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:39:31.858380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:39:31.858484 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:39:31.858549 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.858530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl" (OuterVolumeSpecName: "kube-api-access-5fspl") pod "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" (UID: "56a1afb4-e8f1-4896-9d8a-37fe1212cd4f"). InnerVolumeSpecName "kube-api-access-5fspl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:39:31.957202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957160    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.957202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957195    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-kubeconfig\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.957202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957205    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-cacerts\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.957434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957215    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-local-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.957434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957225    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fspl\" (UniqueName: \"kubernetes.io/projected/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-kube-api-access-5fspl\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:31.957434 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:31.957234    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f-istio-csr-dns-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:39:32.087827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.087734    2577 generic.go:356] "Generic (PLEG): container finished" podID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerID="c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3" exitCode=0
Sep 16 00:39:32.087827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.087777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" event={"ID":"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f","Type":"ContainerDied","Data":"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"}
Sep 16 00:39:32.087827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.087792    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"
Sep 16 00:39:32.087827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.087827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb" event={"ID":"56a1afb4-e8f1-4896-9d8a-37fe1212cd4f","Type":"ContainerDied","Data":"14eb7e27cca74c25a683a111c651bbde063e6ce9c2085ae68eb876e16ef8982a"}
Sep 16 00:39:32.088159 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.087859    2577 scope.go:117] "RemoveContainer" containerID="c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"
Sep 16 00:39:32.098843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.098814    2577 scope.go:117] "RemoveContainer" containerID="c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"
Sep 16 00:39:32.099181 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:39:32.099156    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3\": container with ID starting with c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3 not found: ID does not exist" containerID="c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"
Sep 16 00:39:32.099278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.099188    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3"} err="failed to get container status \"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3\": rpc error: code = NotFound desc = could not find container \"c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3\": container with ID starting with c105a6b86317ceccd11153cd7781348ee2de480f2874a27f394cd93ecc1c19c3 not found: ID does not exist"
Sep 16 00:39:32.111592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.111564    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:32.116044 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.116019    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8jwhb"]
Sep 16 00:39:32.333765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:32.333726    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" path="/var/lib/kubelet/pods/56a1afb4-e8f1-4896-9d8a-37fe1212cd4f/volumes"
Sep 16 00:39:33.094282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:33.094241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp" event={"ID":"ea02c116-d193-437b-9841-e97747a744e6","Type":"ContainerStarted","Data":"5ca50a06f459d6e48178259c0a3ab2bd2da8e999a4e1f52ac7dcc94fcdc6bcd1"}
Sep 16 00:39:33.094740 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:33.094358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:39:33.117241 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:33.117103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp" podStartSLOduration=1.416920291 podStartE2EDuration="4.117080107s" totalImagesPullingTime="2.700159816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:33.115303271 +0000 UTC m=+343.276813544" watchObservedRunningTime="2026-09-16 00:39:33.117080107 +0000 UTC m=+343.278590382"
Sep 16 00:39:35.105565 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:35.105521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb" event={"ID":"4946dd9c-c8b5-4e77-a870-12138bdb5102","Type":"ContainerStarted","Data":"99bfeb701d97abf83fbdcd4533399d4d52ee42c13706165a058c5750d1de1dd1"}
Sep 16 00:39:35.106101 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:35.105693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:39:35.121644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:39:35.121592    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb" podStartSLOduration=2.407713836 podStartE2EDuration="6.121576751s" totalImagesPullingTime="3.713862915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:39:35.120254705 +0000 UTC m=+345.281764990" watchObservedRunningTime="2026-09-16 00:39:35.121576751 +0000 UTC m=+345.283087023"
Sep 16 00:40:04.103186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:04.103151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v6wsp"
Sep 16 00:40:06.115684 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:06.115650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7gtb"
Sep 16 00:40:36.905095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.905056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"]
Sep 16 00:40:36.905574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.905372    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerName="discovery"
Sep 16 00:40:36.905574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.905384    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerName="discovery"
Sep 16 00:40:36.905574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.905461    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="56a1afb4-e8f1-4896-9d8a-37fe1212cd4f" containerName="discovery"
Sep 16 00:40:36.908855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.908837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:36.911771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.911746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 00:40:36.911890 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.911793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 00:40:36.911890 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.911871    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xlkvv\""
Sep 16 00:40:36.912010 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.911895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 00:40:36.918865 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:36.918827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"]
Sep 16 00:40:37.022204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:37.022162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwkf\" (UniqueName: \"kubernetes.io/projected/bd0ab4c9-5929-4726-82d5-b32d1ce186f5-kube-api-access-2nwkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-dldsr\" (UID: \"bd0ab4c9-5929-4726-82d5-b32d1ce186f5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:37.122745 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:37.122708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwkf\" (UniqueName: \"kubernetes.io/projected/bd0ab4c9-5929-4726-82d5-b32d1ce186f5-kube-api-access-2nwkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-dldsr\" (UID: \"bd0ab4c9-5929-4726-82d5-b32d1ce186f5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:37.131399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:37.131372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwkf\" (UniqueName: \"kubernetes.io/projected/bd0ab4c9-5929-4726-82d5-b32d1ce186f5-kube-api-access-2nwkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-dldsr\" (UID: \"bd0ab4c9-5929-4726-82d5-b32d1ce186f5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:37.220451 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:37.220340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:37.500320 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:37.500295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"]
Sep 16 00:40:37.503302 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:40:37.503270    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0ab4c9_5929_4726_82d5_b32d1ce186f5.slice/crio-fab9c1df5e642bba65e5f28bd8a93c9142bbdfee4c0dc59296f7e55c7aecb58d WatchSource:0}: Error finding container fab9c1df5e642bba65e5f28bd8a93c9142bbdfee4c0dc59296f7e55c7aecb58d: Status 404 returned error can't find the container with id fab9c1df5e642bba65e5f28bd8a93c9142bbdfee4c0dc59296f7e55c7aecb58d
Sep 16 00:40:38.399721 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:38.399671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr" event={"ID":"bd0ab4c9-5929-4726-82d5-b32d1ce186f5","Type":"ContainerStarted","Data":"fab9c1df5e642bba65e5f28bd8a93c9142bbdfee4c0dc59296f7e55c7aecb58d"}
Sep 16 00:40:40.411886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.411789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr" event={"ID":"bd0ab4c9-5929-4726-82d5-b32d1ce186f5","Type":"ContainerStarted","Data":"0be38712ec71336364a0bdf6ca3a6617af65faf9e50bd01527fb63775d22266f"}
Sep 16 00:40:40.412290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.411902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:40.429678 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.429621    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr" podStartSLOduration=1.865484161 podStartE2EDuration="4.42960534s" totalImagesPullingTime="2.564121179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:40:40.427936698 +0000 UTC m=+410.589446969" watchObservedRunningTime="2026-09-16 00:40:40.42960534 +0000 UTC m=+410.591115611"
Sep 16 00:40:40.508765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.508727    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pbbrm"]
Sep 16 00:40:40.512424 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.512408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:40.514793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.514773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j4n9z\""
Sep 16 00:40:40.523187 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.523164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pbbrm"]
Sep 16 00:40:40.554419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.554381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmnd\" (UniqueName: \"kubernetes.io/projected/f66545bd-d6b9-4daf-af99-260078e24d84-kube-api-access-7bmnd\") pod \"authorino-operator-76b478bd78-pbbrm\" (UID: \"f66545bd-d6b9-4daf-af99-260078e24d84\") " pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:40.655380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.655342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmnd\" (UniqueName: \"kubernetes.io/projected/f66545bd-d6b9-4daf-af99-260078e24d84-kube-api-access-7bmnd\") pod \"authorino-operator-76b478bd78-pbbrm\" (UID: \"f66545bd-d6b9-4daf-af99-260078e24d84\") " pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:40.664572 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.664502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmnd\" (UniqueName: \"kubernetes.io/projected/f66545bd-d6b9-4daf-af99-260078e24d84-kube-api-access-7bmnd\") pod \"authorino-operator-76b478bd78-pbbrm\" (UID: \"f66545bd-d6b9-4daf-af99-260078e24d84\") " pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:40.822710 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:40.822672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:41.099803 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:41.099777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pbbrm"]
Sep 16 00:40:41.102012 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:40:41.101985    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66545bd_d6b9_4daf_af99_260078e24d84.slice/crio-2080e053ea0ed2a303e6f7954ae468d0218a4137535cfb89a7269ee85e1a1f5f WatchSource:0}: Error finding container 2080e053ea0ed2a303e6f7954ae468d0218a4137535cfb89a7269ee85e1a1f5f: Status 404 returned error can't find the container with id 2080e053ea0ed2a303e6f7954ae468d0218a4137535cfb89a7269ee85e1a1f5f
Sep 16 00:40:41.417843 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:41.417753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm" event={"ID":"f66545bd-d6b9-4daf-af99-260078e24d84","Type":"ContainerStarted","Data":"2080e053ea0ed2a303e6f7954ae468d0218a4137535cfb89a7269ee85e1a1f5f"}
Sep 16 00:40:42.903012 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:42.902980    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"]
Sep 16 00:40:42.907418 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:42.907398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:42.910123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:42.910099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jnnr4\""
Sep 16 00:40:42.919110 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:42.919080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"]
Sep 16 00:40:42.977805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:42.977767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr5q\" (UniqueName: \"kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q\") pod \"limitador-operator-controller-manager-74495c69f8-74z6f\" (UID: \"e7488072-49d0-4986-8674-b85b5a681d55\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:43.078826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.078785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr5q\" (UniqueName: \"kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q\") pod \"limitador-operator-controller-manager-74495c69f8-74z6f\" (UID: \"e7488072-49d0-4986-8674-b85b5a681d55\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:43.091545 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.091516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr5q\" (UniqueName: \"kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q\") pod \"limitador-operator-controller-manager-74495c69f8-74z6f\" (UID: \"e7488072-49d0-4986-8674-b85b5a681d55\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:43.218488 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.218386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:43.429628 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.429597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm" event={"ID":"f66545bd-d6b9-4daf-af99-260078e24d84","Type":"ContainerStarted","Data":"69063f03c68272e5f3741ac7f1f1545eaa1fd8a55ea3e092b6e75614758035f0"}
Sep 16 00:40:43.429826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.429777    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:43.465387 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.465328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm" podStartSLOduration=1.715799142 podStartE2EDuration="3.465311006s" totalImagesPullingTime="1.749511864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:40:43.463354766 +0000 UTC m=+413.624865038" watchObservedRunningTime="2026-09-16 00:40:43.465311006 +0000 UTC m=+413.626821278"
Sep 16 00:40:43.508427 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:43.508354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"]
Sep 16 00:40:43.510841 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:40:43.510813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7488072_49d0_4986_8674_b85b5a681d55.slice/crio-a134702733a4b622d6d19b2f5bf49eb6f78c4c93d3869fa74e92c6ff0b44b67b WatchSource:0}: Error finding container a134702733a4b622d6d19b2f5bf49eb6f78c4c93d3869fa74e92c6ff0b44b67b: Status 404 returned error can't find the container with id a134702733a4b622d6d19b2f5bf49eb6f78c4c93d3869fa74e92c6ff0b44b67b
Sep 16 00:40:44.436849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:44.436801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f" event={"ID":"e7488072-49d0-4986-8674-b85b5a681d55","Type":"ContainerStarted","Data":"a134702733a4b622d6d19b2f5bf49eb6f78c4c93d3869fa74e92c6ff0b44b67b"}
Sep 16 00:40:45.443264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:45.443173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f" event={"ID":"e7488072-49d0-4986-8674-b85b5a681d55","Type":"ContainerStarted","Data":"b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5"}
Sep 16 00:40:45.443264 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:45.443247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:40:45.471086 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:45.471013    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f" podStartSLOduration=1.896608921 podStartE2EDuration="3.470993902s" totalImagesPullingTime="1.574384981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:40:45.469888104 +0000 UTC m=+415.631398368" watchObservedRunningTime="2026-09-16 00:40:45.470993902 +0000 UTC m=+415.632504173"
Sep 16 00:40:47.195003 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.194965    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"]
Sep 16 00:40:47.198748 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.198726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.202246 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.202224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8hbc7\""
Sep 16 00:40:47.213237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.213212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"]
Sep 16 00:40:47.316005 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.315929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.316186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.316056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntpq\" (UniqueName: \"kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.417433 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.417380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.417612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.417473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntpq\" (UniqueName: \"kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.417876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.417850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.426794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.426764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntpq\" (UniqueName: \"kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.508933 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.508902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:47.784703 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:47.784668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"]
Sep 16 00:40:47.787274 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:40:47.787247    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3109a4_0a19_4596_bcb7_e50199e46938.slice/crio-513405d953e9e069b933ead02fd053003da0e6d8dea1b35d8f4dbc2ae033611c WatchSource:0}: Error finding container 513405d953e9e069b933ead02fd053003da0e6d8dea1b35d8f4dbc2ae033611c: Status 404 returned error can't find the container with id 513405d953e9e069b933ead02fd053003da0e6d8dea1b35d8f4dbc2ae033611c
Sep 16 00:40:48.458318 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:48.458285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz" event={"ID":"cf3109a4-0a19-4596-bcb7-e50199e46938","Type":"ContainerStarted","Data":"513405d953e9e069b933ead02fd053003da0e6d8dea1b35d8f4dbc2ae033611c"}
Sep 16 00:40:51.421569 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:51.421525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dldsr"
Sep 16 00:40:52.483315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:52.483221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz" event={"ID":"cf3109a4-0a19-4596-bcb7-e50199e46938","Type":"ContainerStarted","Data":"b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927"}
Sep 16 00:40:52.483729 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:52.483362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:40:52.503408 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:52.503353    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz" podStartSLOduration=1.142688813 podStartE2EDuration="5.50333703s" totalImagesPullingTime="4.360648217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:40:52.501781359 +0000 UTC m=+422.663291632" watchObservedRunningTime="2026-09-16 00:40:52.50333703 +0000 UTC m=+422.664847303"
Sep 16 00:40:54.439449 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:54.439417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-pbbrm"
Sep 16 00:40:56.450310 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:40:56.450281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:03.490644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:03.490609    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.613724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.613680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"]
Sep 16 00:41:05.614249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.614044    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" containerName="manager" containerID="cri-o://b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927" gracePeriod=2
Sep 16 00:41:05.622356 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.622329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:05.629278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.629249    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"]
Sep 16 00:41:05.629390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.629359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.641633 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.641590    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cf3109a4-0a19-4596-bcb7-e50199e46938\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\": pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.644514 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.644486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:05.646369 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.646329    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.646783 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.646764    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"]
Sep 16 00:41:05.647014 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.646978    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f" podUID="e7488072-49d0-4986-8674-b85b5a681d55" containerName="manager" containerID="cri-o://b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5" gracePeriod=2
Sep 16 00:41:05.655090 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.655057    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"]
Sep 16 00:41:05.671394 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.671350    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.673150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673128    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"]
Sep 16 00:41:05.673657 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7488072-49d0-4986-8674-b85b5a681d55" containerName="manager"
Sep 16 00:41:05.673723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7488072-49d0-4986-8674-b85b5a681d55" containerName="manager"
Sep 16 00:41:05.673723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673677    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" containerName="manager"
Sep 16 00:41:05.673723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" containerName="manager"
Sep 16 00:41:05.673857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673788    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7488072-49d0-4986-8674-b85b5a681d55" containerName="manager"
Sep 16 00:41:05.673857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.673804    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" containerName="manager"
Sep 16 00:41:05.674914 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.674842    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74z6f\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="e7488072-49d0-4986-8674-b85b5a681d55" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:05.677643 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.677625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:05.679796 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.679763    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74z6f\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="e7488072-49d0-4986-8674-b85b5a681d55" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:05.688463 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.688386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"]
Sep 16 00:41:05.704191 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.704158    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.772608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.772553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.772608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.772605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m524c\" (UniqueName: \"kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.772888 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.772633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5rv\" (UniqueName: \"kubernetes.io/projected/e45e50df-6c82-4f5b-8d9f-a59596e928a6-kube-api-access-2s5rv\") pod \"limitador-operator-controller-manager-74495c69f8-7wlls\" (UID: \"e45e50df-6c82-4f5b-8d9f-a59596e928a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:05.873938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.873906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.873938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.873964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m524c\" (UniqueName: \"kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.874153 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.873994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5rv\" (UniqueName: \"kubernetes.io/projected/e45e50df-6c82-4f5b-8d9f-a59596e928a6-kube-api-access-2s5rv\") pod \"limitador-operator-controller-manager-74495c69f8-7wlls\" (UID: \"e45e50df-6c82-4f5b-8d9f-a59596e928a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:05.874297 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.874272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.882761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.882736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5rv\" (UniqueName: \"kubernetes.io/projected/e45e50df-6c82-4f5b-8d9f-a59596e928a6-kube-api-access-2s5rv\") pod \"limitador-operator-controller-manager-74495c69f8-7wlls\" (UID: \"e45e50df-6c82-4f5b-8d9f-a59596e928a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:05.883425 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.883402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m524c\" (UniqueName: \"kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-qt2gf\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:05.885532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.885516    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.887635 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.887611    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74z6f\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="e7488072-49d0-4986-8674-b85b5a681d55" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:05.889506 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.889483    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.897939 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.897916    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:05.899983 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.899957    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-74z6f\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="e7488072-49d0-4986-8674-b85b5a681d55" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:05.901803 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:05.901780    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5v4mz\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:05.975284 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.975252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qntpq\" (UniqueName: \"kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq\") pod \"cf3109a4-0a19-4596-bcb7-e50199e46938\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") "
Sep 16 00:41:05.975456 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.975318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hr5q\" (UniqueName: \"kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q\") pod \"e7488072-49d0-4986-8674-b85b5a681d55\" (UID: \"e7488072-49d0-4986-8674-b85b5a681d55\") "
Sep 16 00:41:05.975456 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.975340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume\") pod \"cf3109a4-0a19-4596-bcb7-e50199e46938\" (UID: \"cf3109a4-0a19-4596-bcb7-e50199e46938\") "
Sep 16 00:41:05.975872 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.975834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cf3109a4-0a19-4596-bcb7-e50199e46938" (UID: "cf3109a4-0a19-4596-bcb7-e50199e46938"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:41:05.977702 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.977676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q" (OuterVolumeSpecName: "kube-api-access-2hr5q") pod "e7488072-49d0-4986-8674-b85b5a681d55" (UID: "e7488072-49d0-4986-8674-b85b5a681d55"). InnerVolumeSpecName "kube-api-access-2hr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:41:05.977773 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:05.977679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq" (OuterVolumeSpecName: "kube-api-access-qntpq") pod "cf3109a4-0a19-4596-bcb7-e50199e46938" (UID: "cf3109a4-0a19-4596-bcb7-e50199e46938"). InnerVolumeSpecName "kube-api-access-qntpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:41:06.063042 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.063003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:06.070401 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.070380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:06.076903 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.076872    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hr5q\" (UniqueName: \"kubernetes.io/projected/e7488072-49d0-4986-8674-b85b5a681d55-kube-api-access-2hr5q\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:41:06.076997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.076908    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf3109a4-0a19-4596-bcb7-e50199e46938-extensions-socket-volume\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:41:06.076997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.076922    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntpq\" (UniqueName: \"kubernetes.io/projected/cf3109a4-0a19-4596-bcb7-e50199e46938-kube-api-access-qntpq\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:41:06.333871 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.333840    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3109a4-0a19-4596-bcb7-e50199e46938" path="/var/lib/kubelet/pods/cf3109a4-0a19-4596-bcb7-e50199e46938/volumes"
Sep 16 00:41:06.334236 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.334221    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7488072-49d0-4986-8674-b85b5a681d55" path="/var/lib/kubelet/pods/e7488072-49d0-4986-8674-b85b5a681d55/volumes"
Sep 16 00:41:06.346490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.346470    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:06.349370 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:41:06.349344    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cda583_7518_41d1_8aae_a31981aa43d5.slice/crio-88fd635cded962d510cb4bf96d14d2bf164d9d88e153ed63bdec7aed43084ec0 WatchSource:0}: Error finding container 88fd635cded962d510cb4bf96d14d2bf164d9d88e153ed63bdec7aed43084ec0: Status 404 returned error can't find the container with id 88fd635cded962d510cb4bf96d14d2bf164d9d88e153ed63bdec7aed43084ec0
Sep 16 00:41:06.352525 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.352493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"]
Sep 16 00:41:06.356515 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:41:06.356495    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45e50df_6c82_4f5b_8d9f_a59596e928a6.slice/crio-406697563c76e3b1bc557814c35b8f59d32bfa27d85e837a8d460350e4c619de WatchSource:0}: Error finding container 406697563c76e3b1bc557814c35b8f59d32bfa27d85e837a8d460350e4c619de: Status 404 returned error can't find the container with id 406697563c76e3b1bc557814c35b8f59d32bfa27d85e837a8d460350e4c619de
Sep 16 00:41:06.551533 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.551497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls" event={"ID":"e45e50df-6c82-4f5b-8d9f-a59596e928a6","Type":"ContainerStarted","Data":"89cb646759008bcfcd9d240acf07a36eb940cf8e916835d07f15971633f86a4c"}
Sep 16 00:41:06.551731 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.551565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:06.551731 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.551591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls" event={"ID":"e45e50df-6c82-4f5b-8d9f-a59596e928a6","Type":"ContainerStarted","Data":"406697563c76e3b1bc557814c35b8f59d32bfa27d85e837a8d460350e4c619de"}
Sep 16 00:41:06.552926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.552903    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf3109a4-0a19-4596-bcb7-e50199e46938" containerID="b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927" exitCode=0
Sep 16 00:41:06.553008 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.552945    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5v4mz"
Sep 16 00:41:06.553008 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.552998    2577 scope.go:117] "RemoveContainer" containerID="b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927"
Sep 16 00:41:06.554397 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.554376    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7488072-49d0-4986-8674-b85b5a681d55" containerID="b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5" exitCode=0
Sep 16 00:41:06.554526 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.554453    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-74z6f"
Sep 16 00:41:06.557259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.557203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" event={"ID":"d9cda583-7518-41d1-8aae-a31981aa43d5","Type":"ContainerStarted","Data":"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"}
Sep 16 00:41:06.557431 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.557406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" event={"ID":"d9cda583-7518-41d1-8aae-a31981aa43d5","Type":"ContainerStarted","Data":"88fd635cded962d510cb4bf96d14d2bf164d9d88e153ed63bdec7aed43084ec0"}
Sep 16 00:41:06.562904 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.562879    2577 scope.go:117] "RemoveContainer" containerID="b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927"
Sep 16 00:41:06.563215 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:06.563189    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927\": container with ID starting with b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927 not found: ID does not exist" containerID="b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927"
Sep 16 00:41:06.563258 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.563223    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927"} err="failed to get container status \"b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927\": rpc error: code = NotFound desc = could not find container \"b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927\": container with ID starting with b31b0ea876cbca5d27ffde1e505d2967136ce1a2f5cc9b23a9d3b2f5770ad927 not found: ID does not exist"
Sep 16 00:41:06.563258 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.563241    2577 scope.go:117] "RemoveContainer" containerID="b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5"
Sep 16 00:41:06.570928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.570863    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls" podStartSLOduration=1.5708417940000001 podStartE2EDuration="1.570841794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:41:06.568685362 +0000 UTC m=+436.730195648" watchObservedRunningTime="2026-09-16 00:41:06.570841794 +0000 UTC m=+436.732352069"
Sep 16 00:41:06.572280 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.572259    2577 scope.go:117] "RemoveContainer" containerID="b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5"
Sep 16 00:41:06.572590 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:06.572571    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5\": container with ID starting with b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5 not found: ID does not exist" containerID="b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5"
Sep 16 00:41:06.572671 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.572598    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5"} err="failed to get container status \"b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5\": rpc error: code = NotFound desc = could not find container \"b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5\": container with ID starting with b284ab8e5d68feedccc75616955e7c0b8eb17067beef48ee9ae9a070873f8ed5 not found: ID does not exist"
Sep 16 00:41:06.590825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:06.590774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" podStartSLOduration=1.590757092 podStartE2EDuration="1.590757092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:41:06.58998005 +0000 UTC m=+436.751490319" watchObservedRunningTime="2026-09-16 00:41:06.590757092 +0000 UTC m=+436.752267363"
Sep 16 00:41:07.564791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:07.564749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:17.567472 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:17.567343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7wlls"
Sep 16 00:41:18.572641 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:18.572612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:30.626172 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.626138    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:30.626715 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.626389    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" podUID="d9cda583-7518-41d1-8aae-a31981aa43d5" containerName="manager" containerID="cri-o://ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9" gracePeriod=10
Sep 16 00:41:30.871715 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.871689    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:30.892272 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.892142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m524c\" (UniqueName: \"kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c\") pod \"d9cda583-7518-41d1-8aae-a31981aa43d5\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") "
Sep 16 00:41:30.892272 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.892187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume\") pod \"d9cda583-7518-41d1-8aae-a31981aa43d5\" (UID: \"d9cda583-7518-41d1-8aae-a31981aa43d5\") "
Sep 16 00:41:30.892650 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.892618    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d9cda583-7518-41d1-8aae-a31981aa43d5" (UID: "d9cda583-7518-41d1-8aae-a31981aa43d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:41:30.894634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.894604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c" (OuterVolumeSpecName: "kube-api-access-m524c") pod "d9cda583-7518-41d1-8aae-a31981aa43d5" (UID: "d9cda583-7518-41d1-8aae-a31981aa43d5"). InnerVolumeSpecName "kube-api-access-m524c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:41:30.992887 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.992847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m524c\" (UniqueName: \"kubernetes.io/projected/d9cda583-7518-41d1-8aae-a31981aa43d5-kube-api-access-m524c\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:41:30.992887 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:30.992878    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cda583-7518-41d1-8aae-a31981aa43d5-extensions-socket-volume\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:41:31.685134 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.685088    2577 generic.go:356] "Generic (PLEG): container finished" podID="d9cda583-7518-41d1-8aae-a31981aa43d5" containerID="ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9" exitCode=0
Sep 16 00:41:31.685134 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.685133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"
Sep 16 00:41:31.685609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.685176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" event={"ID":"d9cda583-7518-41d1-8aae-a31981aa43d5","Type":"ContainerDied","Data":"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"}
Sep 16 00:41:31.685609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.685230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf" event={"ID":"d9cda583-7518-41d1-8aae-a31981aa43d5","Type":"ContainerDied","Data":"88fd635cded962d510cb4bf96d14d2bf164d9d88e153ed63bdec7aed43084ec0"}
Sep 16 00:41:31.685609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.685255    2577 scope.go:117] "RemoveContainer" containerID="ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"
Sep 16 00:41:31.694208 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.694188    2577 scope.go:117] "RemoveContainer" containerID="ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"
Sep 16 00:41:31.694523 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:41:31.694507    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9\": container with ID starting with ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9 not found: ID does not exist" containerID="ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"
Sep 16 00:41:31.694568 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.694531    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9"} err="failed to get container status \"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9\": rpc error: code = NotFound desc = could not find container \"ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9\": container with ID starting with ed42955333cb8b638181a41375766910c99452a36a8c95497733660797808de9 not found: ID does not exist"
Sep 16 00:41:31.707122 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.707094    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:31.709727 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:31.709703    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-qt2gf"]
Sep 16 00:41:32.333174 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:32.333143    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9cda583-7518-41d1-8aae-a31981aa43d5" path="/var/lib/kubelet/pods/d9cda583-7518-41d1-8aae-a31981aa43d5/volumes"
Sep 16 00:41:34.974785 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.974739    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"]
Sep 16 00:41:34.975234 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.975102    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cda583-7518-41d1-8aae-a31981aa43d5" containerName="manager"
Sep 16 00:41:34.975234 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.975114    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cda583-7518-41d1-8aae-a31981aa43d5" containerName="manager"
Sep 16 00:41:34.975234 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.975193    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cda583-7518-41d1-8aae-a31981aa43d5" containerName="manager"
Sep 16 00:41:34.977847 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.977826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:34.980623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.980602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 00:41:34.980757 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.980605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6mxhx\""
Sep 16 00:41:34.989638 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:34.989615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"]
Sep 16 00:41:35.021738 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021738 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.021995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.021976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.022177 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.022000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc65b\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-kube-api-access-sc65b\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.022177 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.022043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a77ed0e-564e-463e-947e-d6501435fbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123024 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.122942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc65b\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-kube-api-access-sc65b\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a77ed0e-564e-463e-947e-d6501435fbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123774 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.123829 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.123786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.124071 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.124009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a77ed0e-564e-463e-947e-d6501435fbb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.125522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.125493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.125636 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.125617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.131318 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.131295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.131417 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.131400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc65b\" (UniqueName: \"kubernetes.io/projected/8a77ed0e-564e-463e-947e-d6501435fbb1-kube-api-access-sc65b\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6m5tb\" (UID: \"8a77ed0e-564e-463e-947e-d6501435fbb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.289597 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.289565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:35.561661 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.561636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"]
Sep 16 00:41:35.563530 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:41:35.563501    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a77ed0e_564e_463e_947e_d6501435fbb1.slice/crio-69edb0dd06fd87f7dc6200a5ed1de266d5c313459de290f16b46963f4a2f7df6 WatchSource:0}: Error finding container 69edb0dd06fd87f7dc6200a5ed1de266d5c313459de290f16b46963f4a2f7df6: Status 404 returned error can't find the container with id 69edb0dd06fd87f7dc6200a5ed1de266d5c313459de290f16b46963f4a2f7df6
Sep 16 00:41:35.707718 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:35.707683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" event={"ID":"8a77ed0e-564e-463e-947e-d6501435fbb1","Type":"ContainerStarted","Data":"69edb0dd06fd87f7dc6200a5ed1de266d5c313459de290f16b46963f4a2f7df6"}
Sep 16 00:41:38.523265 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:38.523225    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:41:38.523530 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:38.523303    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:41:38.523530 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:38.523332    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:41:38.725383 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:38.725336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" event={"ID":"8a77ed0e-564e-463e-947e-d6501435fbb1","Type":"ContainerStarted","Data":"c16cf0671627c2ffa0f4af54eeef29dba32e4371774563811b483d0503138d64"}
Sep 16 00:41:38.745729 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:38.745680    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podStartSLOduration=1.788261007 podStartE2EDuration="4.745660752s" totalImagesPullingTime="2.957399745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:41:38.744044487 +0000 UTC m=+468.905554749" watchObservedRunningTime="2026-09-16 00:41:38.745660752 +0000 UTC m=+468.907171023"
Sep 16 00:41:39.290473 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:39.290416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:39.295228 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:39.295198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:39.731016 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:39.730903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:39.731975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:39.731926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:41:48.613269 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.613231    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:41:48.616668 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.616643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.619186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.619162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6z2cb\""
Sep 16 00:41:48.619289 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.619190    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 00:41:48.625806 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.625781    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:41:48.716249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.716212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:41:48.744779 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.744741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bdb\" (UniqueName: \"kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.744988 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.744804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.845764 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.845730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.845943 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.845824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bdb\" (UniqueName: \"kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.846529 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.846508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.853441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.853406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bdb\" (UniqueName: \"kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb\") pod \"limitador-limitador-8588749977-lx256\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") " pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:48.927414 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:48.927327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:49.200864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:49.200834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:41:49.202879 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:41:49.202835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1d63e4_e88d_40e2_a415_3552b767f2c1.slice/crio-f1ebf84cab91b7a1782faaf5fb576290fe229b6a8f14d466a96e5814369bd78c WatchSource:0}: Error finding container f1ebf84cab91b7a1782faaf5fb576290fe229b6a8f14d466a96e5814369bd78c: Status 404 returned error can't find the container with id f1ebf84cab91b7a1782faaf5fb576290fe229b6a8f14d466a96e5814369bd78c
Sep 16 00:41:49.780617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:49.780585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lx256" event={"ID":"1a1d63e4-e88d-40e2-a415-3552b767f2c1","Type":"ContainerStarted","Data":"f1ebf84cab91b7a1782faaf5fb576290fe229b6a8f14d466a96e5814369bd78c"}
Sep 16 00:41:53.802290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:53.802181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lx256" event={"ID":"1a1d63e4-e88d-40e2-a415-3552b767f2c1","Type":"ContainerStarted","Data":"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"}
Sep 16 00:41:53.802687 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:53.802377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:41:53.818455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:41:53.818403    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lx256" podStartSLOduration=1.48325338 podStartE2EDuration="5.818385276s" totalImagesPullingTime="4.335131896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:41:53.816778929 +0000 UTC m=+483.978289201" watchObservedRunningTime="2026-09-16 00:41:53.818385276 +0000 UTC m=+483.979895611"
Sep 16 00:42:03.257285 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.257249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:42:03.257726 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.257455    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lx256" podUID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" containerName="limitador" containerID="cri-o://a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688" gracePeriod=30
Sep 16 00:42:03.258345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.258083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:42:03.807092 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.807057    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:42:03.853978 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.853860    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" containerID="a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688" exitCode=0
Sep 16 00:42:03.853978 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.853911    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lx256"
Sep 16 00:42:03.853978 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.853963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lx256" event={"ID":"1a1d63e4-e88d-40e2-a415-3552b767f2c1","Type":"ContainerDied","Data":"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"}
Sep 16 00:42:03.854188 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.854048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lx256" event={"ID":"1a1d63e4-e88d-40e2-a415-3552b767f2c1","Type":"ContainerDied","Data":"f1ebf84cab91b7a1782faaf5fb576290fe229b6a8f14d466a96e5814369bd78c"}
Sep 16 00:42:03.854188 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.854072    2577 scope.go:117] "RemoveContainer" containerID="a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"
Sep 16 00:42:03.862375 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.862356    2577 scope.go:117] "RemoveContainer" containerID="a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"
Sep 16 00:42:03.862652 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:03.862635    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688\": container with ID starting with a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688 not found: ID does not exist" containerID="a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"
Sep 16 00:42:03.862693 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.862663    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688"} err="failed to get container status \"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688\": rpc error: code = NotFound desc = could not find container \"a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688\": container with ID starting with a8fca3d57694b61cfe8181a652837d1f50c6cd129803b1499ece8f259add9688 not found: ID does not exist"
Sep 16 00:42:03.886574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.886543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file\") pod \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") "
Sep 16 00:42:03.886746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.886608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2bdb\" (UniqueName: \"kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb\") pod \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\" (UID: \"1a1d63e4-e88d-40e2-a415-3552b767f2c1\") "
Sep 16 00:42:03.886928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.886900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file" (OuterVolumeSpecName: "config-file") pod "1a1d63e4-e88d-40e2-a415-3552b767f2c1" (UID: "1a1d63e4-e88d-40e2-a415-3552b767f2c1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:42:03.888909 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.888887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb" (OuterVolumeSpecName: "kube-api-access-j2bdb") pod "1a1d63e4-e88d-40e2-a415-3552b767f2c1" (UID: "1a1d63e4-e88d-40e2-a415-3552b767f2c1"). InnerVolumeSpecName "kube-api-access-j2bdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:42:03.987317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.987273    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a1d63e4-e88d-40e2-a415-3552b767f2c1-config-file\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:03.987317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:03.987306    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bdb\" (UniqueName: \"kubernetes.io/projected/1a1d63e4-e88d-40e2-a415-3552b767f2c1-kube-api-access-j2bdb\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:04.174294 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:04.174237    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:42:04.175876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:04.175850    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lx256"]
Sep 16 00:42:04.334714 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:04.334680    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" path="/var/lib/kubelet/pods/1a1d63e4-e88d-40e2-a415-3552b767f2c1/volumes"
Sep 16 00:42:06.678294 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.678250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"]
Sep 16 00:42:06.678818 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.678802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" containerName="limitador"
Sep 16 00:42:06.678924 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.678822    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" containerName="limitador"
Sep 16 00:42:06.678980 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.678922    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1d63e4-e88d-40e2-a415-3552b767f2c1" containerName="limitador"
Sep 16 00:42:06.684005 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.683981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.687360 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.687321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ggxbp\""
Sep 16 00:42:06.687360 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.687321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 00:42:06.687360 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.687358    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 00:42:06.687601 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.687326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 00:42:06.688978 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.688930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"]
Sep 16 00:42:06.812747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.812710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f17ca288-36ad-4af0-a87e-2e493f159081-data\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.812747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.812752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6nc\" (UniqueName: \"kubernetes.io/projected/f17ca288-36ad-4af0-a87e-2e493f159081-kube-api-access-5h6nc\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.913859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.913817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f17ca288-36ad-4af0-a87e-2e493f159081-data\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.913859 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.913859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6nc\" (UniqueName: \"kubernetes.io/projected/f17ca288-36ad-4af0-a87e-2e493f159081-kube-api-access-5h6nc\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.914284 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.914260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f17ca288-36ad-4af0-a87e-2e493f159081-data\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.922087 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.922055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6nc\" (UniqueName: \"kubernetes.io/projected/f17ca288-36ad-4af0-a87e-2e493f159081-kube-api-access-5h6nc\") pod \"postgres-69fb4594cf-2qbhw\" (UID: \"f17ca288-36ad-4af0-a87e-2e493f159081\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:06.997517 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:06.997414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:07.267912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:07.267887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"]
Sep 16 00:42:07.270370 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:42:07.270336    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17ca288_36ad_4af0_a87e_2e493f159081.slice/crio-13aef034435e31f320425423a84003549739e4d78b8578e7f36cd37fe1cef29e WatchSource:0}: Error finding container 13aef034435e31f320425423a84003549739e4d78b8578e7f36cd37fe1cef29e: Status 404 returned error can't find the container with id 13aef034435e31f320425423a84003549739e4d78b8578e7f36cd37fe1cef29e
Sep 16 00:42:07.878222 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:07.878176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw" event={"ID":"f17ca288-36ad-4af0-a87e-2e493f159081","Type":"ContainerStarted","Data":"13aef034435e31f320425423a84003549739e4d78b8578e7f36cd37fe1cef29e"}
Sep 16 00:42:12.910300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:12.910257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw" event={"ID":"f17ca288-36ad-4af0-a87e-2e493f159081","Type":"ContainerStarted","Data":"e9d125b24519bf1d7548bfb55bcc3b4b22041248475c46f2e903a4e60c17b8e2"}
Sep 16 00:42:12.910723 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:12.910338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:12.925807 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:12.925750    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw" podStartSLOduration=1.882152746 podStartE2EDuration="6.925731852s" totalImagesPullingTime="5.043579106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:42:12.924497956 +0000 UTC m=+503.086008232" watchObservedRunningTime="2026-09-16 00:42:12.925731852 +0000 UTC m=+503.087242124"
Sep 16 00:42:18.945118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:18.945087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2qbhw"
Sep 16 00:42:19.492623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.492582    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-7n45z"]
Sep 16 00:42:19.501864 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.501827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:19.503745 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.503715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-7n45z"]
Sep 16 00:42:19.504317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.504296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mc4dw\""
Sep 16 00:42:19.619821 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.619783    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7n45z"]
Sep 16 00:42:19.620096 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:19.620074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qqbhh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-7n45z" podUID="15a14894-c9e6-43e3-b69e-eda3b32cb1fd"
Sep 16 00:42:19.629382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.629340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbhh\" (UniqueName: \"kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh\") pod \"authorino-557f876f69-7n45z\" (UID: \"15a14894-c9e6-43e3-b69e-eda3b32cb1fd\") " pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:19.641409 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.641372    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fj6nq"]
Sep 16 00:42:19.654281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.654242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fj6nq"]
Sep 16 00:42:19.654492 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.654340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.656967 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.656919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 00:42:19.730045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.729991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbhh\" (UniqueName: \"kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh\") pod \"authorino-557f876f69-7n45z\" (UID: \"15a14894-c9e6-43e3-b69e-eda3b32cb1fd\") " pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:19.738249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.738205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbhh\" (UniqueName: \"kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh\") pod \"authorino-557f876f69-7n45z\" (UID: \"15a14894-c9e6-43e3-b69e-eda3b32cb1fd\") " pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:19.754328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.750504    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fj6nq"]
Sep 16 00:42:19.754328 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:19.751103    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dgmph tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-fj6nq" podUID="339cd863-3bd6-4e4b-9584-987f6572f3a1"
Sep 16 00:42:19.831077 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.831041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.831331 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.831119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmph\" (UniqueName: \"kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.931906 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.931864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.932133 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.932002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmph\" (UniqueName: \"kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.934663 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.934628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.939813 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.939788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmph\" (UniqueName: \"kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph\") pod \"authorino-6d4dfc649f-fj6nq\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") " pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.942815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.942791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:19.942984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:19.942963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:20.000278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.000244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:20.015374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.015283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:20.133555 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.133515    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgmph\" (UniqueName: \"kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph\") pod \"339cd863-3bd6-4e4b-9584-987f6572f3a1\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") "
Sep 16 00:42:20.133736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.133623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert\") pod \"339cd863-3bd6-4e4b-9584-987f6572f3a1\" (UID: \"339cd863-3bd6-4e4b-9584-987f6572f3a1\") "
Sep 16 00:42:20.133736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.133660    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqbhh\" (UniqueName: \"kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh\") pod \"15a14894-c9e6-43e3-b69e-eda3b32cb1fd\" (UID: \"15a14894-c9e6-43e3-b69e-eda3b32cb1fd\") "
Sep 16 00:42:20.135911 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.135883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "339cd863-3bd6-4e4b-9584-987f6572f3a1" (UID: "339cd863-3bd6-4e4b-9584-987f6572f3a1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:42:20.136061 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.136014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh" (OuterVolumeSpecName: "kube-api-access-qqbhh") pod "15a14894-c9e6-43e3-b69e-eda3b32cb1fd" (UID: "15a14894-c9e6-43e3-b69e-eda3b32cb1fd"). InnerVolumeSpecName "kube-api-access-qqbhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:42:20.136061 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.136039    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph" (OuterVolumeSpecName: "kube-api-access-dgmph") pod "339cd863-3bd6-4e4b-9584-987f6572f3a1" (UID: "339cd863-3bd6-4e4b-9584-987f6572f3a1"). InnerVolumeSpecName "kube-api-access-dgmph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:42:20.234975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.234914    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/339cd863-3bd6-4e4b-9584-987f6572f3a1-tls-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:20.234975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.234945    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbhh\" (UniqueName: \"kubernetes.io/projected/15a14894-c9e6-43e3-b69e-eda3b32cb1fd-kube-api-access-qqbhh\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:20.234975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.234982    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmph\" (UniqueName: \"kubernetes.io/projected/339cd863-3bd6-4e4b-9584-987f6572f3a1-kube-api-access-dgmph\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:20.949841 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.949809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7n45z"
Sep 16 00:42:20.950103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.949811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fj6nq"
Sep 16 00:42:20.976672 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.976632    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fj6nq"]
Sep 16 00:42:20.981275 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.981235    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fj6nq"]
Sep 16 00:42:20.998921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:20.998876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7n45z"]
Sep 16 00:42:21.002623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:21.002593    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-7n45z"]
Sep 16 00:42:22.333527 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:22.333496    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a14894-c9e6-43e3-b69e-eda3b32cb1fd" path="/var/lib/kubelet/pods/15a14894-c9e6-43e3-b69e-eda3b32cb1fd/volumes"
Sep 16 00:42:22.333912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:22.333752    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="339cd863-3bd6-4e4b-9584-987f6572f3a1" path="/var/lib/kubelet/pods/339cd863-3bd6-4e4b-9584-987f6572f3a1/volumes"
Sep 16 00:42:26.649693 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.649650    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:26.657632 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.657593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.660313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.660248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 00:42:26.660313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.660310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 00:42:26.660505 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.660363    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk8jt\""
Sep 16 00:42:26.660645 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.660624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 00:42:26.663985 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.663944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:26.688081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.688038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.688271 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.688102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.688271 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.688129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjz8p\" (UniqueName: \"kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.788886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.788843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.789127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.788908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.789127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.788931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjz8p\" (UniqueName: \"kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.789127 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.789038    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 16 00:42:26.789127 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.789066    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 16 00:42:26.789127 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.789128    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert podName:33124861-b85f-46ec-a4ce-083bee82b8ee nodeName:}" failed. No retries permitted until 2026-09-16 00:42:27.289104219 +0000 UTC m=+517.450614474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert") pod "maas-controller-75dc6cbdf-t2xdg" (UID: "33124861-b85f-46ec-a4ce-083bee82b8ee") : secret "maas-controller-webhook-cert" not found
Sep 16 00:42:26.789409 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.789184    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs podName:33124861-b85f-46ec-a4ce-083bee82b8ee nodeName:}" failed. No retries permitted until 2026-09-16 00:42:27.289166679 +0000 UTC m=+517.450676947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs") pod "maas-controller-75dc6cbdf-t2xdg" (UID: "33124861-b85f-46ec-a4ce-083bee82b8ee") : secret "maas-controller-metrics-tls" not found
Sep 16 00:42:26.798253 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.798225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjz8p\" (UniqueName: \"kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:26.821479 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.821436    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:42:26.824203 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.824178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.832719 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.832683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:42:26.890435 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.890391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.890435 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.890443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q67p\" (UniqueName: \"kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.890660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.890519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.991548 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.991449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.991548 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.991545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.991752 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.991579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q67p\" (UniqueName: \"kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:26.991752 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.991677    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 16 00:42:26.991838 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:26.991754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs podName:d57b7bc6-68e1-4d73-8e2d-955ade315358 nodeName:}" failed. No retries permitted until 2026-09-16 00:42:27.491734062 +0000 UTC m=+517.653244331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs") pod "maas-controller-7dbf7685b-94tg9" (UID: "d57b7bc6-68e1-4d73-8e2d-955ade315358") : secret "maas-controller-metrics-tls" not found
Sep 16 00:42:26.994363 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.994330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:27.000011 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:26.999984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q67p\" (UniqueName: \"kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:27.294047 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.294009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:27.294217 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.294066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:27.296721 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.296688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:27.296845 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.296795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") pod \"maas-controller-75dc6cbdf-t2xdg\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") " pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:27.496341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.496299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:27.498883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.498844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") pod \"maas-controller-7dbf7685b-94tg9\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") " pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:27.570316 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.570229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:27.737079 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.737045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:27.850328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.850300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:27.852785 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:42:27.852755    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33124861_b85f_46ec_a4ce_083bee82b8ee.slice/crio-38d8db20da8815bc37550af9d0520df98f4069ebbebed0f44109b0c8068370be WatchSource:0}: Error finding container 38d8db20da8815bc37550af9d0520df98f4069ebbebed0f44109b0c8068370be: Status 404 returned error can't find the container with id 38d8db20da8815bc37550af9d0520df98f4069ebbebed0f44109b0c8068370be
Sep 16 00:42:27.984674 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:27.984636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" event={"ID":"33124861-b85f-46ec-a4ce-083bee82b8ee","Type":"ContainerStarted","Data":"38d8db20da8815bc37550af9d0520df98f4069ebbebed0f44109b0c8068370be"}
Sep 16 00:42:28.010108 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:28.010082    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:42:28.012113 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:42:28.012086    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57b7bc6_68e1_4d73_8e2d_955ade315358.slice/crio-2758c17ca3317c4bf5870541428a342e94a2d574a6295d0eccd30b5dc6ba0913 WatchSource:0}: Error finding container 2758c17ca3317c4bf5870541428a342e94a2d574a6295d0eccd30b5dc6ba0913: Status 404 returned error can't find the container with id 2758c17ca3317c4bf5870541428a342e94a2d574a6295d0eccd30b5dc6ba0913
Sep 16 00:42:28.997923 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:28.997853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-94tg9" event={"ID":"d57b7bc6-68e1-4d73-8e2d-955ade315358","Type":"ContainerStarted","Data":"2758c17ca3317c4bf5870541428a342e94a2d574a6295d0eccd30b5dc6ba0913"}
Sep 16 00:42:31.012447 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.012407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" event={"ID":"33124861-b85f-46ec-a4ce-083bee82b8ee","Type":"ContainerStarted","Data":"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"}
Sep 16 00:42:31.012907 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.012487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:31.014164 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.014134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-94tg9" event={"ID":"d57b7bc6-68e1-4d73-8e2d-955ade315358","Type":"ContainerStarted","Data":"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"}
Sep 16 00:42:31.014292 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.014239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:31.031667 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.031591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" podStartSLOduration=2.490483751 podStartE2EDuration="5.031568476s" totalImagesPullingTime="2.541084725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:42:31.030755725 +0000 UTC m=+521.192266010" watchObservedRunningTime="2026-09-16 00:42:31.031568476 +0000 UTC m=+521.193078749"
Sep 16 00:42:31.054646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:31.054595    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dbf7685b-94tg9" podStartSLOduration=2.683011011 podStartE2EDuration="5.054579749s" totalImagesPullingTime="2.371568738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:42:31.051680517 +0000 UTC m=+521.213190782" watchObservedRunningTime="2026-09-16 00:42:31.054579749 +0000 UTC m=+521.216090045"
Sep 16 00:42:42.021459 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.021374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:42:42.021459 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.021445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:42.079432 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.079397    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:42.079642 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.079616    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" podUID="33124861-b85f-46ec-a4ce-083bee82b8ee" containerName="manager" containerID="cri-o://662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5" gracePeriod=30
Sep 16 00:42:42.345591 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.345564    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:42.430257 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.430218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjz8p\" (UniqueName: \"kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p\") pod \"33124861-b85f-46ec-a4ce-083bee82b8ee\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") "
Sep 16 00:42:42.430441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.430324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") pod \"33124861-b85f-46ec-a4ce-083bee82b8ee\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") "
Sep 16 00:42:42.430441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.430355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") pod \"33124861-b85f-46ec-a4ce-083bee82b8ee\" (UID: \"33124861-b85f-46ec-a4ce-083bee82b8ee\") "
Sep 16 00:42:42.432869 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.432840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "33124861-b85f-46ec-a4ce-083bee82b8ee" (UID: "33124861-b85f-46ec-a4ce-083bee82b8ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:42:42.432869 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.432832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33124861-b85f-46ec-a4ce-083bee82b8ee" (UID: "33124861-b85f-46ec-a4ce-083bee82b8ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:42:42.433053 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.432841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p" (OuterVolumeSpecName: "kube-api-access-gjz8p") pod "33124861-b85f-46ec-a4ce-083bee82b8ee" (UID: "33124861-b85f-46ec-a4ce-083bee82b8ee"). InnerVolumeSpecName "kube-api-access-gjz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:42:42.531499 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.531443    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-webhook-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:42.531499 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.531488    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33124861-b85f-46ec-a4ce-083bee82b8ee-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:42.531499 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:42.531501    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjz8p\" (UniqueName: \"kubernetes.io/projected/33124861-b85f-46ec-a4ce-083bee82b8ee-kube-api-access-gjz8p\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:42:43.079380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.079342    2577 generic.go:356] "Generic (PLEG): container finished" podID="33124861-b85f-46ec-a4ce-083bee82b8ee" containerID="662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5" exitCode=0
Sep 16 00:42:43.079802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.079394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg"
Sep 16 00:42:43.079802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.079403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" event={"ID":"33124861-b85f-46ec-a4ce-083bee82b8ee","Type":"ContainerDied","Data":"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"}
Sep 16 00:42:43.079802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.079455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-t2xdg" event={"ID":"33124861-b85f-46ec-a4ce-083bee82b8ee","Type":"ContainerDied","Data":"38d8db20da8815bc37550af9d0520df98f4069ebbebed0f44109b0c8068370be"}
Sep 16 00:42:43.079802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.079486    2577 scope.go:117] "RemoveContainer" containerID="662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"
Sep 16 00:42:43.089278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.089258    2577 scope.go:117] "RemoveContainer" containerID="662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"
Sep 16 00:42:43.089571 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:42:43.089546    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5\": container with ID starting with 662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5 not found: ID does not exist" containerID="662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"
Sep 16 00:42:43.089666 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.089576    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5"} err="failed to get container status \"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5\": rpc error: code = NotFound desc = could not find container \"662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5\": container with ID starting with 662861caf2f88a6be804969666832e299e07ef15af68885794a599e1599395b5 not found: ID does not exist"
Sep 16 00:42:43.100989 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.100944    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:43.102603 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:43.102582    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-t2xdg"]
Sep 16 00:42:44.333485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:42:44.333448    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33124861-b85f-46ec-a4ce-083bee82b8ee" path="/var/lib/kubelet/pods/33124861-b85f-46ec-a4ce-083bee82b8ee/volumes"
Sep 16 00:43:03.932202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.932162    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"]
Sep 16 00:43:03.932607 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.932592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33124861-b85f-46ec-a4ce-083bee82b8ee" containerName="manager"
Sep 16 00:43:03.932607 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.932604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33124861-b85f-46ec-a4ce-083bee82b8ee" containerName="manager"
Sep 16 00:43:03.932720 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.932708    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33124861-b85f-46ec-a4ce-083bee82b8ee" containerName="manager"
Sep 16 00:43:03.940386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.940351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:03.944429 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.944384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 00:43:03.944618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.944459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 00:43:03.944618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.944474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5hv\""
Sep 16 00:43:03.946986 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:03.946943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"]
Sep 16 00:43:04.131412 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.131375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szf6n\" (UniqueName: \"kubernetes.io/projected/01fb386a-c060-476e-b387-df59ab63916f-kube-api-access-szf6n\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.131412 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.131413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb386a-c060-476e-b387-df59ab63916f-openshift-service-ca\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.131653 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.131437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-metrics-certs\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.131653 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.131522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-maas-api-tls\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.235381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.235265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb386a-c060-476e-b387-df59ab63916f-openshift-service-ca\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.235381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.235341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-metrics-certs\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.235623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.235390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-maas-api-tls\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.235623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.235541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szf6n\" (UniqueName: \"kubernetes.io/projected/01fb386a-c060-476e-b387-df59ab63916f-kube-api-access-szf6n\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.236165 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.236138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb386a-c060-476e-b387-df59ab63916f-openshift-service-ca\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.238313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.238279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-maas-api-tls\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.238438 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.238386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb386a-c060-476e-b387-df59ab63916f-metrics-certs\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.243783 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.243744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szf6n\" (UniqueName: \"kubernetes.io/projected/01fb386a-c060-476e-b387-df59ab63916f-kube-api-access-szf6n\") pod \"maas-api-59f6566bbc-ch48s\" (UID: \"01fb386a-c060-476e-b387-df59ab63916f\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.253617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.253580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:04.364784 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.364746    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:43:04.368070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.368051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.370791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.370763    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 00:43:04.370937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.370821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sgjrw\""
Sep 16 00:43:04.377934 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.377905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:43:04.394452 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.394418    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:43:04.397470 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.397442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.407569 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.407360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:43:04.438119 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.438084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.438300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.438147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgr86\" (UniqueName: \"kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.438300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.438190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.438300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.438230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvv4l\" (UniqueName: \"kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.532324 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.532286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"]
Sep 16 00:43:04.536564 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:43:04.536530    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fb386a_c060_476e_b387_df59ab63916f.slice/crio-35d894a66b5a42a2b1d291643958024577bd66970f95e448fc5f159fb1b27bea WatchSource:0}: Error finding container 35d894a66b5a42a2b1d291643958024577bd66970f95e448fc5f159fb1b27bea: Status 404 returned error can't find the container with id 35d894a66b5a42a2b1d291643958024577bd66970f95e448fc5f159fb1b27bea
Sep 16 00:43:04.539078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.539049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.539182 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.539095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgr86\" (UniqueName: \"kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.539182 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.539113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.539182 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.539145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvv4l\" (UniqueName: \"kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.540038 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.540009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.540038 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.540029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.549155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.549125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgr86\" (UniqueName: \"kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86\") pod \"payload-pre-processing-59fbc47bf8-2sgt5\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.549290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.549126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvv4l\" (UniqueName: \"kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l\") pod \"payload-processing-5b54b7fbb7-w7rs4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.681265 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.681224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:04.713284 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.713250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:04.968330 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.968242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:43:04.972592 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:43:04.972560    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577f8656_8d3f_4965_b1b8_b1c0aea831ce.slice/crio-3394a6a852f3bfa1b23d4186bf2bf67fe4a466c072def1e561d5c4e734066888 WatchSource:0}: Error finding container 3394a6a852f3bfa1b23d4186bf2bf67fe4a466c072def1e561d5c4e734066888: Status 404 returned error can't find the container with id 3394a6a852f3bfa1b23d4186bf2bf67fe4a466c072def1e561d5c4e734066888
Sep 16 00:43:04.996849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:04.996821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:43:04.999231 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:43:04.999197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796bdbf4_e595_439d_b392_e565b91b44f4.slice/crio-275c9168c95646f8303257bda7b7071cc580fc67273972bdbd814747d160197e WatchSource:0}: Error finding container 275c9168c95646f8303257bda7b7071cc580fc67273972bdbd814747d160197e: Status 404 returned error can't find the container with id 275c9168c95646f8303257bda7b7071cc580fc67273972bdbd814747d160197e
Sep 16 00:43:05.190518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:05.190477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" event={"ID":"796bdbf4-e595-439d-b392-e565b91b44f4","Type":"ContainerStarted","Data":"275c9168c95646f8303257bda7b7071cc580fc67273972bdbd814747d160197e"}
Sep 16 00:43:05.191771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:05.191746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s" event={"ID":"01fb386a-c060-476e-b387-df59ab63916f","Type":"ContainerStarted","Data":"35d894a66b5a42a2b1d291643958024577bd66970f95e448fc5f159fb1b27bea"}
Sep 16 00:43:05.192850 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:05.192829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" event={"ID":"577f8656-8d3f-4965-b1b8-b1c0aea831ce","Type":"ContainerStarted","Data":"3394a6a852f3bfa1b23d4186bf2bf67fe4a466c072def1e561d5c4e734066888"}
Sep 16 00:43:09.221143 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.221061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" event={"ID":"577f8656-8d3f-4965-b1b8-b1c0aea831ce","Type":"ContainerStarted","Data":"c440f4854ee129a6bc1b8c0f6551e07847857e6d08399521cd5682520e0f3381"}
Sep 16 00:43:09.221657 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.221167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:09.223105 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.223078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" event={"ID":"796bdbf4-e595-439d-b392-e565b91b44f4","Type":"ContainerStarted","Data":"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"}
Sep 16 00:43:09.223238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.223223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:09.225501 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.225475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s" event={"ID":"01fb386a-c060-476e-b387-df59ab63916f","Type":"ContainerStarted","Data":"6f001c8829b7826a732c033d469a2597b3e61d089a9fb5f53a03961e1ac3d603"}
Sep 16 00:43:09.225631 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.225606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:09.238397 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.238346    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" podStartSLOduration=1.642804806 podStartE2EDuration="5.23833064s" totalImagesPullingTime="3.595525834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:09.236834291 +0000 UTC m=+559.398344563" watchObservedRunningTime="2026-09-16 00:43:09.23833064 +0000 UTC m=+559.399840956"
Sep 16 00:43:09.273865 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.273805    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" podStartSLOduration=1.735271101 podStartE2EDuration="5.273783965s" totalImagesPullingTime="3.538512864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:09.27147605 +0000 UTC m=+559.432986322" watchObservedRunningTime="2026-09-16 00:43:09.273783965 +0000 UTC m=+559.435294237"
Sep 16 00:43:09.274270 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:09.274238    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s" podStartSLOduration=2.337313994 podStartE2EDuration="6.274229186s" totalImagesPullingTime="3.936915192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:09.255394794 +0000 UTC m=+559.416905067" watchObservedRunningTime="2026-09-16 00:43:09.274229186 +0000 UTC m=+559.435739458"
Sep 16 00:43:14.257416 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.257373    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerID="c16cf0671627c2ffa0f4af54eeef29dba32e4371774563811b483d0503138d64" exitCode=0
Sep 16 00:43:14.258065 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.257455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" event={"ID":"8a77ed0e-564e-463e-947e-d6501435fbb1","Type":"ContainerDied","Data":"c16cf0671627c2ffa0f4af54eeef29dba32e4371774563811b483d0503138d64"}
Sep 16 00:43:14.258161 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.258086    2577 scope.go:117] "RemoveContainer" containerID="c16cf0671627c2ffa0f4af54eeef29dba32e4371774563811b483d0503138d64"
Sep 16 00:43:14.259418 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.259381    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:43:14.259518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.259505    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:43:14.259557 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:14.259540    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:43:15.238491 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:15.238447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ch48s"
Sep 16 00:43:15.270735 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:15.270682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" event={"ID":"8a77ed0e-564e-463e-947e-d6501435fbb1","Type":"ContainerStarted","Data":"a5887289047384bc52e8e251e53cbc9971f378c222f97c8969729ed2c892c2bb"}
Sep 16 00:43:15.290485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:15.290442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:43:15.290735 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:15.290710    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:15.290804 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:15.290769    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:16.290135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:16.290086    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:16.290606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:16.290154    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:17.290565 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:17.290520    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:17.291095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:17.290593    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:18.290423 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:18.290383    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:18.290623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:18.290460    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:19.290383 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:19.290346    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:19.290612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:19.290403    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:20.234925 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:20.234890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:43:20.235323 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:20.235193    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:43:20.290192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:20.290119    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6m5tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 16 00:43:20.290192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:20.290188    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb" podUID="8a77ed0e-564e-463e-947e-d6501435fbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 16 00:43:21.294765 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:21.294726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:43:21.295256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:21.295001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:43:21.295970 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:21.295928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6m5tb"
Sep 16 00:43:34.688807 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.688767    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"]
Sep 16 00:43:34.692404 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.692377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.695245 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.695219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lbjkk\""
Sep 16 00:43:34.695463 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.695446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 00:43:34.696371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.696353    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 00:43:34.696458 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.696379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 00:43:34.703213 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.703189    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"]
Sep 16 00:43:34.765120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtcq\" (UniqueName: \"kubernetes.io/projected/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-kube-api-access-hvtcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.765295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.765295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.765295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.765295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.765295 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.765288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866362 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtcq\" (UniqueName: \"kubernetes.io/projected/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-kube-api-access-hvtcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866640 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866684 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866831 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.866917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.867000 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.866941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.868855 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.868831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.869275 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.869254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:34.874112 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:34.874090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtcq\" (UniqueName: \"kubernetes.io/projected/aa8fdffd-f2a2-4ccd-be3d-b09e45269346-kube-api-access-hvtcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42\" (UID: \"aa8fdffd-f2a2-4ccd-be3d-b09e45269346\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:35.004493 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:35.004455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:35.280243 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:35.280217    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"]
Sep 16 00:43:35.281921 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:43:35.281895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8fdffd_f2a2_4ccd_be3d_b09e45269346.slice/crio-7d16f428c8ffb56d1b33bab8885b446662a134e74f78d849173628c7c15d73ff WatchSource:0}: Error finding container 7d16f428c8ffb56d1b33bab8885b446662a134e74f78d849173628c7c15d73ff: Status 404 returned error can't find the container with id 7d16f428c8ffb56d1b33bab8885b446662a134e74f78d849173628c7c15d73ff
Sep 16 00:43:35.377970 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:35.377927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42" event={"ID":"aa8fdffd-f2a2-4ccd-be3d-b09e45269346","Type":"ContainerStarted","Data":"7d16f428c8ffb56d1b33bab8885b446662a134e74f78d849173628c7c15d73ff"}
Sep 16 00:43:37.389554 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:37.389521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42" event={"ID":"aa8fdffd-f2a2-4ccd-be3d-b09e45269346","Type":"ContainerStarted","Data":"4ef32ea5980599750fd2e0253f562b49208bde77b8d4f071d49f0f08b70673a6"}
Sep 16 00:43:37.407201 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:37.407147    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42" podStartSLOduration=2.298944435 podStartE2EDuration="3.407130343s" totalImagesPullingTime="1.108185908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:43:37.405357681 +0000 UTC m=+587.566867957" watchObservedRunningTime="2026-09-16 00:43:37.407130343 +0000 UTC m=+587.568640612"
Sep 16 00:43:45.005116 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:45.005066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:45.005645 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:45.005157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:45.017764 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:45.017739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:45.441644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:45.441556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-68d42"
Sep 16 00:43:50.309493 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:50.309462    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:43:50.310801 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:43:50.310778    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:44:44.892636 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:44.892597    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"]
Sep 16 00:44:44.901572 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:44.901532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:44.905631 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:44.905596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 00:44:44.907518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:44.907492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"]
Sep 16 00:44:45.005912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.005867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.006127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.005984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.006127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.006023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.006127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.006090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.006302 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.006182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgg6\" (UniqueName: \"kubernetes.io/projected/66c436b4-fc93-4bc0-873c-968f94a3f23f-kube-api-access-9xgg6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.006302 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.006231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c436b4-fc93-4bc0-873c-968f94a3f23f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.106911 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.106872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.106944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgg6\" (UniqueName: \"kubernetes.io/projected/66c436b4-fc93-4bc0-873c-968f94a3f23f-kube-api-access-9xgg6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c436b4-fc93-4bc0-873c-968f94a3f23f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107403 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107493 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.107921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.107857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.109886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.109861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c436b4-fc93-4bc0-873c-968f94a3f23f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.110106 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.110081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c436b4-fc93-4bc0-873c-968f94a3f23f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.114689 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.114657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgg6\" (UniqueName: \"kubernetes.io/projected/66c436b4-fc93-4bc0-873c-968f94a3f23f-kube-api-access-9xgg6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6\" (UID: \"66c436b4-fc93-4bc0-873c-968f94a3f23f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.213946 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.213843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:45.487801 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.487769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"]
Sep 16 00:44:45.489783 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:44:45.489743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c436b4_fc93_4bc0_873c_968f94a3f23f.slice/crio-c9288f3e8415ad28281a5176146e1f07c1319aeb78c7c547b7ff6dcb3721e3ea WatchSource:0}: Error finding container c9288f3e8415ad28281a5176146e1f07c1319aeb78c7c547b7ff6dcb3721e3ea: Status 404 returned error can't find the container with id c9288f3e8415ad28281a5176146e1f07c1319aeb78c7c547b7ff6dcb3721e3ea
Sep 16 00:44:45.491205 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.491184    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:44:45.745643 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:45.745562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6" event={"ID":"66c436b4-fc93-4bc0-873c-968f94a3f23f","Type":"ContainerStarted","Data":"c9288f3e8415ad28281a5176146e1f07c1319aeb78c7c547b7ff6dcb3721e3ea"}
Sep 16 00:44:46.751846 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:46.751813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6" event={"ID":"66c436b4-fc93-4bc0-873c-968f94a3f23f","Type":"ContainerStarted","Data":"b95d8d093d1e117113a700d35e16244b6a6ca52568d08cd6356857c21612ded9"}
Sep 16 00:44:46.768265 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:46.768214    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6" podStartSLOduration=2.499085811 podStartE2EDuration="2.768198874s" totalImagesPullingTime="269.113063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:44:46.766665355 +0000 UTC m=+656.928175628" watchObservedRunningTime="2026-09-16 00:44:46.768198874 +0000 UTC m=+656.929709145"
Sep 16 00:44:55.214873 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:55.214821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:55.215320 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:55.214905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:55.227760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:55.227735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:44:55.817139 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:44:55.817102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8pbg6"
Sep 16 00:46:33.697066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:33.697024    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:46:33.697551 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:33.697261    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dbf7685b-94tg9" podUID="d57b7bc6-68e1-4d73-8e2d-955ade315358" containerName="manager" containerID="cri-o://fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa" gracePeriod=30
Sep 16 00:46:33.959736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:33.959665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:46:33.963129 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:33.963097    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:46:33.963321 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:33.963295    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" podUID="796bdbf4-e595-439d-b392-e565b91b44f4" containerName="payload-processing" containerID="cri-o://ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5" gracePeriod=30
Sep 16 00:46:34.093736 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.093618    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:46:34.093917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.093871    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" podUID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" containerName="payload-pre-processing" containerID="cri-o://c440f4854ee129a6bc1b8c0f6551e07847857e6d08399521cd5682520e0f3381" gracePeriod=30
Sep 16 00:46:34.110608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.110239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q67p\" (UniqueName: \"kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p\") pod \"d57b7bc6-68e1-4d73-8e2d-955ade315358\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") "
Sep 16 00:46:34.110608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.110309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") pod \"d57b7bc6-68e1-4d73-8e2d-955ade315358\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") "
Sep 16 00:46:34.110608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.110395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert\") pod \"d57b7bc6-68e1-4d73-8e2d-955ade315358\" (UID: \"d57b7bc6-68e1-4d73-8e2d-955ade315358\") "
Sep 16 00:46:34.113598 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.113514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d57b7bc6-68e1-4d73-8e2d-955ade315358" (UID: "d57b7bc6-68e1-4d73-8e2d-955ade315358"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:46:34.113821 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.113793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p" (OuterVolumeSpecName: "kube-api-access-7q67p") pod "d57b7bc6-68e1-4d73-8e2d-955ade315358" (UID: "d57b7bc6-68e1-4d73-8e2d-955ade315358"). InnerVolumeSpecName "kube-api-access-7q67p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:46:34.113937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.113919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d57b7bc6-68e1-4d73-8e2d-955ade315358" (UID: "d57b7bc6-68e1-4d73-8e2d-955ade315358"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:46:34.211977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.211904    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q67p\" (UniqueName: \"kubernetes.io/projected/d57b7bc6-68e1-4d73-8e2d-955ade315358-kube-api-access-7q67p\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.211977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.211931    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.211977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.211942    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d57b7bc6-68e1-4d73-8e2d-955ade315358-webhook-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.212690 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.212669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:46:34.312762 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.312731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume\") pod \"796bdbf4-e595-439d-b392-e565b91b44f4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") "
Sep 16 00:46:34.312979 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.312861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvv4l\" (UniqueName: \"kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l\") pod \"796bdbf4-e595-439d-b392-e565b91b44f4\" (UID: \"796bdbf4-e595-439d-b392-e565b91b44f4\") "
Sep 16 00:46:34.313718 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.313659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "796bdbf4-e595-439d-b392-e565b91b44f4" (UID: "796bdbf4-e595-439d-b392-e565b91b44f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:46:34.314572 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.314519    2577 generic.go:356] "Generic (PLEG): container finished" podID="796bdbf4-e595-439d-b392-e565b91b44f4" containerID="ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5" exitCode=0
Sep 16 00:46:34.314698 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.314682    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"
Sep 16 00:46:34.314791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.314702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" event={"ID":"796bdbf4-e595-439d-b392-e565b91b44f4","Type":"ContainerDied","Data":"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"}
Sep 16 00:46:34.314791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.314763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7rs4" event={"ID":"796bdbf4-e595-439d-b392-e565b91b44f4","Type":"ContainerDied","Data":"275c9168c95646f8303257bda7b7071cc580fc67273972bdbd814747d160197e"}
Sep 16 00:46:34.314791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.314797    2577 scope.go:117] "RemoveContainer" containerID="ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"
Sep 16 00:46:34.316471 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.316443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l" (OuterVolumeSpecName: "kube-api-access-pvv4l") pod "796bdbf4-e595-439d-b392-e565b91b44f4" (UID: "796bdbf4-e595-439d-b392-e565b91b44f4"). InnerVolumeSpecName "kube-api-access-pvv4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:46:34.316771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.316747    2577 generic.go:356] "Generic (PLEG): container finished" podID="d57b7bc6-68e1-4d73-8e2d-955ade315358" containerID="fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa" exitCode=0
Sep 16 00:46:34.316882 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.316824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-94tg9" event={"ID":"d57b7bc6-68e1-4d73-8e2d-955ade315358","Type":"ContainerDied","Data":"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"}
Sep 16 00:46:34.316935 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.316893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-94tg9" event={"ID":"d57b7bc6-68e1-4d73-8e2d-955ade315358","Type":"ContainerDied","Data":"2758c17ca3317c4bf5870541428a342e94a2d574a6295d0eccd30b5dc6ba0913"}
Sep 16 00:46:34.316935 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.316907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-94tg9"
Sep 16 00:46:34.318753 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.318727    2577 generic.go:356] "Generic (PLEG): container finished" podID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" containerID="c440f4854ee129a6bc1b8c0f6551e07847857e6d08399521cd5682520e0f3381" exitCode=0
Sep 16 00:46:34.318881 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.318747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" event={"ID":"577f8656-8d3f-4965-b1b8-b1c0aea831ce","Type":"ContainerDied","Data":"c440f4854ee129a6bc1b8c0f6551e07847857e6d08399521cd5682520e0f3381"}
Sep 16 00:46:34.327154 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.326989    2577 scope.go:117] "RemoveContainer" containerID="ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"
Sep 16 00:46:34.327364 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:46:34.327332    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5\": container with ID starting with ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5 not found: ID does not exist" containerID="ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"
Sep 16 00:46:34.327471 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.327376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5"} err="failed to get container status \"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5\": rpc error: code = NotFound desc = could not find container \"ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5\": container with ID starting with ffaa8c71454e0e8b3b5b4bf21e2fc013a3b94ffd6ea74b812a9539e571de24b5 not found: ID does not exist"
Sep 16 00:46:34.327471 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.327401    2577 scope.go:117] "RemoveContainer" containerID="fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"
Sep 16 00:46:34.340340 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.339968    2577 scope.go:117] "RemoveContainer" containerID="fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"
Sep 16 00:46:34.340673 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:46:34.340644    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa\": container with ID starting with fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa not found: ID does not exist" containerID="fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"
Sep 16 00:46:34.340768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.340689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa"} err="failed to get container status \"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa\": rpc error: code = NotFound desc = could not find container \"fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa\": container with ID starting with fa07c86a44154a0228851bae0b0ee7cf31e91cea4b7ff57dde55f0ec7d2b80fa not found: ID does not exist"
Sep 16 00:46:34.344177 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.344150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:46:34.349081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.348921    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-94tg9"]
Sep 16 00:46:34.357965 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.357929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:46:34.414325 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.414284    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796bdbf4-e595-439d-b392-e565b91b44f4-plugins-config-volume\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.414325 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.414319    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvv4l\" (UniqueName: \"kubernetes.io/projected/796bdbf4-e595-439d-b392-e565b91b44f4-kube-api-access-pvv4l\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.514608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.514572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgr86\" (UniqueName: \"kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86\") pod \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") "
Sep 16 00:46:34.514805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.514699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume\") pod \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\" (UID: \"577f8656-8d3f-4965-b1b8-b1c0aea831ce\") "
Sep 16 00:46:34.515191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.515166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "577f8656-8d3f-4965-b1b8-b1c0aea831ce" (UID: "577f8656-8d3f-4965-b1b8-b1c0aea831ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:46:34.517029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.516938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86" (OuterVolumeSpecName: "kube-api-access-wgr86") pod "577f8656-8d3f-4965-b1b8-b1c0aea831ce" (UID: "577f8656-8d3f-4965-b1b8-b1c0aea831ce"). InnerVolumeSpecName "kube-api-access-wgr86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:46:34.615540 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.615504    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgr86\" (UniqueName: \"kubernetes.io/projected/577f8656-8d3f-4965-b1b8-b1c0aea831ce-kube-api-access-wgr86\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.615540 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.615535    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577f8656-8d3f-4965-b1b8-b1c0aea831ce-plugins-config-volume\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:46:34.633442 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.633341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:46:34.635567 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.635541    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7rs4"]
Sep 16 00:46:34.827137 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827096    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 00:46:34.827676 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827646    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" containerName="payload-pre-processing"
Sep 16 00:46:34.827676 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827675    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" containerName="payload-pre-processing"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827700    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796bdbf4-e595-439d-b392-e565b91b44f4" containerName="payload-processing"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="796bdbf4-e595-439d-b392-e565b91b44f4" containerName="payload-processing"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827723    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d57b7bc6-68e1-4d73-8e2d-955ade315358" containerName="manager"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827728    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d57b7bc6-68e1-4d73-8e2d-955ade315358" containerName="manager"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827814    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d57b7bc6-68e1-4d73-8e2d-955ade315358" containerName="manager"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827830    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" containerName="payload-pre-processing"
Sep 16 00:46:34.827857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.827840    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="796bdbf4-e595-439d-b392-e565b91b44f4" containerName="payload-processing"
Sep 16 00:46:34.831120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.831101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:34.833634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.833611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 00:46:34.833634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.833629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 00:46:34.833793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.833631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 00:46:34.833793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.833724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk8jt\""
Sep 16 00:46:34.840779 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.840755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 00:46:34.918200 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.918153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:34.918382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.918234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gfk\" (UniqueName: \"kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:34.918382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:34.918260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.019663 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.019574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.019829 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.019674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.019829 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.019742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gfk\" (UniqueName: \"kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.022502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.022469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.022627 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.022501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.029419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.029388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gfk\" (UniqueName: \"kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk\") pod \"maas-controller-7dbf7685b-xb6cv\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") " pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.142569 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.142527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:35.326306 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.326218    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"
Sep 16 00:46:35.326306 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.326249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5" event={"ID":"577f8656-8d3f-4965-b1b8-b1c0aea831ce","Type":"ContainerDied","Data":"3394a6a852f3bfa1b23d4186bf2bf67fe4a466c072def1e561d5c4e734066888"}
Sep 16 00:46:35.326522 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.326342    2577 scope.go:117] "RemoveContainer" containerID="c440f4854ee129a6bc1b8c0f6551e07847857e6d08399521cd5682520e0f3381"
Sep 16 00:46:35.352101 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.352068    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:46:35.354918 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.354895    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2sgt5"]
Sep 16 00:46:35.416886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:35.416857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 00:46:35.418538 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:46:35.418511    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0108d3a9_9020_4e42_b146_0b9a1f9cb809.slice/crio-96de4f773b23a1598f4711ed8c7f8ee9859b79a9f081c226388e1b9f8627eb62 WatchSource:0}: Error finding container 96de4f773b23a1598f4711ed8c7f8ee9859b79a9f081c226388e1b9f8627eb62: Status 404 returned error can't find the container with id 96de4f773b23a1598f4711ed8c7f8ee9859b79a9f081c226388e1b9f8627eb62
Sep 16 00:46:36.334236 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.334202    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577f8656-8d3f-4965-b1b8-b1c0aea831ce" path="/var/lib/kubelet/pods/577f8656-8d3f-4965-b1b8-b1c0aea831ce/volumes"
Sep 16 00:46:36.334688 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.334641    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796bdbf4-e595-439d-b392-e565b91b44f4" path="/var/lib/kubelet/pods/796bdbf4-e595-439d-b392-e565b91b44f4/volumes"
Sep 16 00:46:36.335073 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.335054    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d57b7bc6-68e1-4d73-8e2d-955ade315358" path="/var/lib/kubelet/pods/d57b7bc6-68e1-4d73-8e2d-955ade315358/volumes"
Sep 16 00:46:36.335558 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.335530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" event={"ID":"0108d3a9-9020-4e42-b146-0b9a1f9cb809","Type":"ContainerStarted","Data":"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"}
Sep 16 00:46:36.335644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.335576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" event={"ID":"0108d3a9-9020-4e42-b146-0b9a1f9cb809","Type":"ContainerStarted","Data":"96de4f773b23a1598f4711ed8c7f8ee9859b79a9f081c226388e1b9f8627eb62"}
Sep 16 00:46:36.335700 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.335649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:36.355503 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:36.355449    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" podStartSLOduration=1.978335591 podStartE2EDuration="2.35543204s" totalImagesPullingTime="377.096449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:46:36.354004421 +0000 UTC m=+766.515514690" watchObservedRunningTime="2026-09-16 00:46:36.35543204 +0000 UTC m=+766.516942312"
Sep 16 00:46:47.343763 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.343729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 00:46:47.626029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.625914    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:46:47.629886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.629859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:47.632438 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.632417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6hbpf\""
Sep 16 00:46:47.632566 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.632419    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 16 00:46:47.636849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.636823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:46:47.735209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.735166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgfg\" (UniqueName: \"kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg\") pod \"ai-gateway-controller-6468cc78b9-4xjzx\" (UID: \"9be7d6fc-3bbc-40bd-944e-5e588a664110\") " pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:47.836073 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.836034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgfg\" (UniqueName: \"kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg\") pod \"ai-gateway-controller-6468cc78b9-4xjzx\" (UID: \"9be7d6fc-3bbc-40bd-944e-5e588a664110\") " pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:47.845276 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.845232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgfg\" (UniqueName: \"kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg\") pod \"ai-gateway-controller-6468cc78b9-4xjzx\" (UID: \"9be7d6fc-3bbc-40bd-944e-5e588a664110\") " pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:47.942876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:47.942754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:48.219004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:48.218977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:46:48.221738 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:46:48.221703    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be7d6fc_3bbc_40bd_944e_5e588a664110.slice/crio-7bc25b66f4857b8aeb514c37149787b7314a94698541b61acdd7ab6993ffe327 WatchSource:0}: Error finding container 7bc25b66f4857b8aeb514c37149787b7314a94698541b61acdd7ab6993ffe327: Status 404 returned error can't find the container with id 7bc25b66f4857b8aeb514c37149787b7314a94698541b61acdd7ab6993ffe327
Sep 16 00:46:48.399349 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:48.399314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" event={"ID":"9be7d6fc-3bbc-40bd-944e-5e588a664110","Type":"ContainerStarted","Data":"7bc25b66f4857b8aeb514c37149787b7314a94698541b61acdd7ab6993ffe327"}
Sep 16 00:46:50.257726 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.257699    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"]
Sep 16 00:46:50.261383 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.261355    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.263937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.263916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 00:46:50.263937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.263926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sgjrw\""
Sep 16 00:46:50.267996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.267934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"]
Sep 16 00:46:50.290066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.290031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"]
Sep 16 00:46:50.293835 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.293808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.304589 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.304559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"]
Sep 16 00:46:50.358454 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.358418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghq78\" (UniqueName: \"kubernetes.io/projected/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-kube-api-access-ghq78\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.358652 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.358490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.411757 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.411716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" event={"ID":"9be7d6fc-3bbc-40bd-944e-5e588a664110","Type":"ContainerStarted","Data":"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"}
Sep 16 00:46:50.412004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.411779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:46:50.428505 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.428445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" podStartSLOduration=1.807275524 podStartE2EDuration="3.428429599s" totalImagesPullingTime="1.621154075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:46:50.427248794 +0000 UTC m=+780.588759067" watchObservedRunningTime="2026-09-16 00:46:50.428429599 +0000 UTC m=+780.589939871"
Sep 16 00:46:50.459815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.459770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghq78\" (UniqueName: \"kubernetes.io/projected/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-kube-api-access-ghq78\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.460051 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.459890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.460051 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.459945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxh6\" (UniqueName: \"kubernetes.io/projected/f0ba643b-242e-49bb-9a71-1746afd71e33-kube-api-access-tbxh6\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.460051 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.460011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba643b-242e-49bb-9a71-1746afd71e33-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.460791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.460769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.468033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.468007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghq78\" (UniqueName: \"kubernetes.io/projected/d7344110-1e8e-40f1-b9e8-c490a1b44d1b-kube-api-access-ghq78\") pod \"payload-processing-5d7dbbd56b-h5j6g\" (UID: \"d7344110-1e8e-40f1-b9e8-c490a1b44d1b\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.560590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.560487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxh6\" (UniqueName: \"kubernetes.io/projected/f0ba643b-242e-49bb-9a71-1746afd71e33-kube-api-access-tbxh6\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.560590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.560552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba643b-242e-49bb-9a71-1746afd71e33-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.561453 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.561433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ba643b-242e-49bb-9a71-1746afd71e33-plugins-config-volume\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.568797 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.568771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxh6\" (UniqueName: \"kubernetes.io/projected/f0ba643b-242e-49bb-9a71-1746afd71e33-kube-api-access-tbxh6\") pod \"payload-pre-processing-6cd864bc95-l9q92\" (UID: \"f0ba643b-242e-49bb-9a71-1746afd71e33\") " pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.574682 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.574650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:50.606516 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.606475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:50.852451 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.852423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"]
Sep 16 00:46:50.854148 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:46:50.854120    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7344110_1e8e_40f1_b9e8_c490a1b44d1b.slice/crio-39d0bc211a2753bd197ca7e04a212e3781b61c1903e46b2ecc47931e642d514a WatchSource:0}: Error finding container 39d0bc211a2753bd197ca7e04a212e3781b61c1903e46b2ecc47931e642d514a: Status 404 returned error can't find the container with id 39d0bc211a2753bd197ca7e04a212e3781b61c1903e46b2ecc47931e642d514a
Sep 16 00:46:50.890248 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:50.890221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"]
Sep 16 00:46:50.892094 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:46:50.892066    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ba643b_242e_49bb_9a71_1746afd71e33.slice/crio-d1330cbadb38e60971da91db38e34d2296c6cfcd33d9f2ef77195404311d5129 WatchSource:0}: Error finding container d1330cbadb38e60971da91db38e34d2296c6cfcd33d9f2ef77195404311d5129: Status 404 returned error can't find the container with id d1330cbadb38e60971da91db38e34d2296c6cfcd33d9f2ef77195404311d5129
Sep 16 00:46:51.420644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:51.420599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" event={"ID":"f0ba643b-242e-49bb-9a71-1746afd71e33","Type":"ContainerStarted","Data":"d1330cbadb38e60971da91db38e34d2296c6cfcd33d9f2ef77195404311d5129"}
Sep 16 00:46:51.422354 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:51.422260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" event={"ID":"d7344110-1e8e-40f1-b9e8-c490a1b44d1b","Type":"ContainerStarted","Data":"39d0bc211a2753bd197ca7e04a212e3781b61c1903e46b2ecc47931e642d514a"}
Sep 16 00:46:52.429771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.429666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" event={"ID":"f0ba643b-242e-49bb-9a71-1746afd71e33","Type":"ContainerStarted","Data":"867ad71baac140a29145eff4efacf2185353d266891e13b3036f5d9c3587d0c0"}
Sep 16 00:46:52.430249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.429809    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:46:52.431425 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.431403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" event={"ID":"d7344110-1e8e-40f1-b9e8-c490a1b44d1b","Type":"ContainerStarted","Data":"6946aaed4976734250e02499ec52cd20004b9163953d412f9fa6d663f8e02c5b"}
Sep 16 00:46:52.431556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.431539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:46:52.446095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.446043    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" podStartSLOduration=1.278772623 podStartE2EDuration="2.446028298s" totalImagesPullingTime="1.167255675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:46:52.444083791 +0000 UTC m=+782.605594075" watchObservedRunningTime="2026-09-16 00:46:52.446028298 +0000 UTC m=+782.607538569"
Sep 16 00:46:52.458925 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:46:52.458862    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" podStartSLOduration=1.251452477 podStartE2EDuration="2.458844688s" totalImagesPullingTime="1.207392211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:46:52.457588912 +0000 UTC m=+782.619099185" watchObservedRunningTime="2026-09-16 00:46:52.458844688 +0000 UTC m=+782.620354959"
Sep 16 00:47:01.424936 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.424902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:47:01.624617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.624577    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-57d6998586-rtxvr"]
Sep 16 00:47:01.629510 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.629487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:01.640693 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.640664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57d6998586-rtxvr"]
Sep 16 00:47:01.667815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.667774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lqr\" (UniqueName: \"kubernetes.io/projected/556b217d-ad1d-4731-a50f-72a315d721d4-kube-api-access-b5lqr\") pod \"ai-gateway-controller-57d6998586-rtxvr\" (UID: \"556b217d-ad1d-4731-a50f-72a315d721d4\") " pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:01.769256 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.769226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lqr\" (UniqueName: \"kubernetes.io/projected/556b217d-ad1d-4731-a50f-72a315d721d4-kube-api-access-b5lqr\") pod \"ai-gateway-controller-57d6998586-rtxvr\" (UID: \"556b217d-ad1d-4731-a50f-72a315d721d4\") " pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:01.778079 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.778051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lqr\" (UniqueName: \"kubernetes.io/projected/556b217d-ad1d-4731-a50f-72a315d721d4-kube-api-access-b5lqr\") pod \"ai-gateway-controller-57d6998586-rtxvr\" (UID: \"556b217d-ad1d-4731-a50f-72a315d721d4\") " pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:01.940278 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:01.940238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:02.212794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:02.212708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57d6998586-rtxvr"]
Sep 16 00:47:02.216219 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:02.216188    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556b217d_ad1d_4731_a50f_72a315d721d4.slice/crio-df6da66113447a33211806458bb678808a5aa7b5efbd2f29ccf45f6486d29863 WatchSource:0}: Error finding container df6da66113447a33211806458bb678808a5aa7b5efbd2f29ccf45f6486d29863: Status 404 returned error can't find the container with id df6da66113447a33211806458bb678808a5aa7b5efbd2f29ccf45f6486d29863
Sep 16 00:47:02.485547 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:02.485451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr" event={"ID":"556b217d-ad1d-4731-a50f-72a315d721d4","Type":"ContainerStarted","Data":"df6da66113447a33211806458bb678808a5aa7b5efbd2f29ccf45f6486d29863"}
Sep 16 00:47:03.438922 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.438885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92"
Sep 16 00:47:03.439138 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.438996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g"
Sep 16 00:47:03.439490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.439471    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:47:03.439641 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.439623    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:47:03.493910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.493871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr" event={"ID":"556b217d-ad1d-4731-a50f-72a315d721d4","Type":"ContainerStarted","Data":"6ec1b5930fec7b7069ed702addcbe25e00f616f06c22292a89376eff5ad1d939"}
Sep 16 00:47:03.494346 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.494078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:03.509814 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:03.509756    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr" podStartSLOduration=2.019417792 podStartE2EDuration="2.50973563s" totalImagesPullingTime="490.317838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:03.509346351 +0000 UTC m=+793.670856623" watchObservedRunningTime="2026-09-16 00:47:03.50973563 +0000 UTC m=+793.671245903"
Sep 16 00:47:14.501595 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.501516    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-57d6998586-rtxvr"
Sep 16 00:47:14.541097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.541059    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:47:14.541303 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.541258    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" podUID="9be7d6fc-3bbc-40bd-944e-5e588a664110" containerName="manager" containerID="cri-o://8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5" gracePeriod=30
Sep 16 00:47:14.793236 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.793206    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:47:14.890977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.890921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvgfg\" (UniqueName: \"kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg\") pod \"9be7d6fc-3bbc-40bd-944e-5e588a664110\" (UID: \"9be7d6fc-3bbc-40bd-944e-5e588a664110\") "
Sep 16 00:47:14.893174 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.893151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg" (OuterVolumeSpecName: "kube-api-access-xvgfg") pod "9be7d6fc-3bbc-40bd-944e-5e588a664110" (UID: "9be7d6fc-3bbc-40bd-944e-5e588a664110"). InnerVolumeSpecName "kube-api-access-xvgfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:47:14.992266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:14.992225    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgfg\" (UniqueName: \"kubernetes.io/projected/9be7d6fc-3bbc-40bd-944e-5e588a664110-kube-api-access-xvgfg\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:47:15.557857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.557818    2577 generic.go:356] "Generic (PLEG): container finished" podID="9be7d6fc-3bbc-40bd-944e-5e588a664110" containerID="8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5" exitCode=0
Sep 16 00:47:15.557857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.557856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"
Sep 16 00:47:15.558374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.557884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" event={"ID":"9be7d6fc-3bbc-40bd-944e-5e588a664110","Type":"ContainerDied","Data":"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"}
Sep 16 00:47:15.558374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.557931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6468cc78b9-4xjzx" event={"ID":"9be7d6fc-3bbc-40bd-944e-5e588a664110","Type":"ContainerDied","Data":"7bc25b66f4857b8aeb514c37149787b7314a94698541b61acdd7ab6993ffe327"}
Sep 16 00:47:15.558374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.557984    2577 scope.go:117] "RemoveContainer" containerID="8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"
Sep 16 00:47:15.567317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.567296    2577 scope.go:117] "RemoveContainer" containerID="8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"
Sep 16 00:47:15.567632 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:15.567609    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5\": container with ID starting with 8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5 not found: ID does not exist" containerID="8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"
Sep 16 00:47:15.567679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.567642    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5"} err="failed to get container status \"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5\": rpc error: code = NotFound desc = could not find container \"8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5\": container with ID starting with 8b9cb65a1c06066a97f33b4ca80d56e3fd991e9c3ce8c2c91d269805c57916f5 not found: ID does not exist"
Sep 16 00:47:15.578356 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.578325    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:47:15.581890 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:15.581861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6468cc78b9-4xjzx"]
Sep 16 00:47:16.333683 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:16.333637    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be7d6fc-3bbc-40bd-944e-5e588a664110" path="/var/lib/kubelet/pods/9be7d6fc-3bbc-40bd-944e-5e588a664110/volumes"
Sep 16 00:47:34.957507 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.957468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:47:34.957996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.957944    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be7d6fc-3bbc-40bd-944e-5e588a664110" containerName="manager"
Sep 16 00:47:34.957996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.957993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be7d6fc-3bbc-40bd-944e-5e588a664110" containerName="manager"
Sep 16 00:47:34.958094 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.958081    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be7d6fc-3bbc-40bd-944e-5e588a664110" containerName="manager"
Sep 16 00:47:34.961752 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.961731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:34.964394 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.964372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-66ecb7-openshift-default-dockercfg-q9qdv\""
Sep 16 00:47:34.973424 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:34.973402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:47:35.024328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.024295    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:47:35.028481 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.028451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.031104 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.031075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0f7c3b-openshift-default-dockercfg-k87m7\""
Sep 16 00:47:35.040171 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.040149    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:47:35.070313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvr5\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.070724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.070641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.171680 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.171680 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvr5\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2s7j\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.171971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.171990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172850 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.172850 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.172755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.173056 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.173032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.173136 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.173023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.174975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.174926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.175234 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.175213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.180007 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.179932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.180279 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.180259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvr5\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5\") pod \"e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") " pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.272929 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.272897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.272937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.272973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.272999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2s7j\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273609 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.273813 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.274061 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.273836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:35.274518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.274492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.275803 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.275780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.276028 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.276008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.283713 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.283689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.284630 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.284610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2s7j\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j\") pod \"e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") " pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.346976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.346618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:35.561702 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.561672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:47:35.563673 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:35.563626    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599cb85a_cb65_4bd6_b651_6abc8e49dcc1.slice/crio-345ba2094d9d02eb0f682b47a0b5ab0218cbad72a867c258a476e5f7c311909f WatchSource:0}: Error finding container 345ba2094d9d02eb0f682b47a0b5ab0218cbad72a867c258a476e5f7c311909f: Status 404 returned error can't find the container with id 345ba2094d9d02eb0f682b47a0b5ab0218cbad72a867c258a476e5f7c311909f
Sep 16 00:47:35.566386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.566347    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.566496 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.566432    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.566496 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.566473    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.639061 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.639015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:47:35.641040 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:35.641002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e41cc9_76ff_4bb7_808b_6a5184ce5f78.slice/crio-51dee0079f398fc4e6abea7de67f1375b38a1f5f25e6e663a09e91d3bcf5e132 WatchSource:0}: Error finding container 51dee0079f398fc4e6abea7de67f1375b38a1f5f25e6e663a09e91d3bcf5e132: Status 404 returned error can't find the container with id 51dee0079f398fc4e6abea7de67f1375b38a1f5f25e6e663a09e91d3bcf5e132
Sep 16 00:47:35.644904 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.644863    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.645066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.645028    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.645338 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.645105    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:35.661212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.661179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerStarted","Data":"51dee0079f398fc4e6abea7de67f1375b38a1f5f25e6e663a09e91d3bcf5e132"}
Sep 16 00:47:35.663355 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.663324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerStarted","Data":"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"}
Sep 16 00:47:35.663488 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.663373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerStarted","Data":"345ba2094d9d02eb0f682b47a0b5ab0218cbad72a867c258a476e5f7c311909f"}
Sep 16 00:47:35.685999 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:35.685911    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podStartSLOduration=1.6858931830000001 podStartE2EDuration="1.685893183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:35.684097401 +0000 UTC m=+825.845607667" watchObservedRunningTime="2026-09-16 00:47:35.685893183 +0000 UTC m=+825.847403455"
Sep 16 00:47:36.274664 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.274625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:36.280686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.280659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:36.670792 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.670690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerStarted","Data":"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"}
Sep 16 00:47:36.671008 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.670983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:36.671760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.671738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:36.689033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:36.688984    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podStartSLOduration=1.688968707 podStartE2EDuration="1.688968707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:36.688661514 +0000 UTC m=+826.850171787" watchObservedRunningTime="2026-09-16 00:47:36.688968707 +0000 UTC m=+826.850478988"
Sep 16 00:47:37.347231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:37.347198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:37.352857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:37.352826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:37.677570 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:37.677483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:37.678726 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:37.678699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:39.032296 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.032262    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:47:39.036147 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.036129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.038556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.038537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-66ecb7-serving-cert\""
Sep 16 00:47:39.047577 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.047553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:47:39.213156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.213124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.213333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.213165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.213333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.213204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.213333 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.213250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7cj\" (UniqueName: \"kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.314554 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.314459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.314554 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.314501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.314554 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.314538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.314851 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.314598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7cj\" (UniqueName: \"kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.314851 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:39.314624    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-serving-cert: secret "maas-api-e2e-inf-a-66ecb7-serving-cert" not found
Sep 16 00:47:39.314851 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:39.314710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls podName:a8330023-ec14-4c23-ba7c-fea1d220636b nodeName:}" failed. No retries permitted until 2026-09-16 00:47:39.814688328 +0000 UTC m=+829.976198597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls") pod "maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" (UID: "a8330023-ec14-4c23-ba7c-fea1d220636b") : secret "maas-api-e2e-inf-a-66ecb7-serving-cert" not found
Sep 16 00:47:39.315400 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.315378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.317373 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.317345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.325485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.325458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7cj\" (UniqueName: \"kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.819413 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.819368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.822076 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.822049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:39.950756 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:39.950724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:40.225153 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.225127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:47:40.227328 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:40.227300    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8330023_ec14_4c23_ba7c_fea1d220636b.slice/crio-6dc8afef4d71b540574bd325a21cf9962af3e971fe7b216bd34e9e3ba45ab7eb WatchSource:0}: Error finding container 6dc8afef4d71b540574bd325a21cf9962af3e971fe7b216bd34e9e3ba45ab7eb: Status 404 returned error can't find the container with id 6dc8afef4d71b540574bd325a21cf9962af3e971fe7b216bd34e9e3ba45ab7eb
Sep 16 00:47:40.510847 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.510812    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:47:40.516692 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.516662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.518972 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.518926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5270e2-serving-cert\""
Sep 16 00:47:40.524307 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.524279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:47:40.626771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.626738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfttw\" (UniqueName: \"kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.626984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.626799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.626984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.626908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.627068 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.627029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.697036 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.697007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" event={"ID":"a8330023-ec14-4c23-ba7c-fea1d220636b","Type":"ContainerStarted","Data":"6dc8afef4d71b540574bd325a21cf9962af3e971fe7b216bd34e9e3ba45ab7eb"}
Sep 16 00:47:40.728210 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.728173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfttw\" (UniqueName: \"kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.728351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.728248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.728351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.728298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.728475 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.728354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.728475 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:40.728400    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-serving-cert: secret "maas-api-e2e-worker-w0-5270e2-serving-cert" not found
Sep 16 00:47:40.728575 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:40.728495    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls podName:ade09380-f47e-4ac4-a994-74e4992fb3ea nodeName:}" failed. No retries permitted until 2026-09-16 00:47:41.228471952 +0000 UTC m=+831.389982215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls") pod "maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" (UID: "ade09380-f47e-4ac4-a994-74e4992fb3ea") : secret "maas-api-e2e-worker-w0-5270e2-serving-cert" not found
Sep 16 00:47:40.729112 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.729047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.731043 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.731021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:40.738756 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:40.738731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfttw\" (UniqueName: \"kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:41.233791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.233753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:41.236781 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.236751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:41.397635 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.397602    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:47:41.402030 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.402001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.404417 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.404380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-0f7c3b-serving-cert\""
Sep 16 00:47:41.411814 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.411786    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:47:41.431883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.431784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:41.537820 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.537782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.538027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.537829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9t4\" (UniqueName: \"kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.538027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.537864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.538027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.537918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.638780 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.638725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.638780 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.638790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9t4\" (UniqueName: \"kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.639097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.638823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.639097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.638885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.639097 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:41.639047    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-serving-cert: secret "maas-api-e2e-worker-w3-0f7c3b-serving-cert" not found
Sep 16 00:47:41.639268 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:47:41.639115    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls podName:c9bae60a-e806-4940-984d-3b090889a55a nodeName:}" failed. No retries permitted until 2026-09-16 00:47:42.139098632 +0000 UTC m=+832.300608882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls") pod "maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" (UID: "c9bae60a-e806-4940-984d-3b090889a55a") : secret "maas-api-e2e-worker-w3-0f7c3b-serving-cert" not found
Sep 16 00:47:41.639591 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.639571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.641560 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.641538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.647420 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.647395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9t4\" (UniqueName: \"kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:41.706136 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.706098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" event={"ID":"a8330023-ec14-4c23-ba7c-fea1d220636b","Type":"ContainerStarted","Data":"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"}
Sep 16 00:47:41.706323 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.706187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:41.711447 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.711426    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:47:41.713222 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:41.713191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade09380_f47e_4ac4_a994_74e4992fb3ea.slice/crio-7c3e7d45a7684d92d7861cd66e9c3842b1eab3fb9c392b6166c721fc272a67a1 WatchSource:0}: Error finding container 7c3e7d45a7684d92d7861cd66e9c3842b1eab3fb9c392b6166c721fc272a67a1: Status 404 returned error can't find the container with id 7c3e7d45a7684d92d7861cd66e9c3842b1eab3fb9c392b6166c721fc272a67a1
Sep 16 00:47:41.723391 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:41.723347    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" podStartSLOduration=2.288887193 podStartE2EDuration="2.723330052s" totalImagesPullingTime="434.442859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:41.721583762 +0000 UTC m=+831.883094036" watchObservedRunningTime="2026-09-16 00:47:41.723330052 +0000 UTC m=+831.884840323"
Sep 16 00:47:42.144323 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.144206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:42.147517 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.147493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:42.317315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.317273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:42.598301 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.598275    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:47:42.600611 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:47:42.600581    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bae60a_e806_4940_984d_3b090889a55a.slice/crio-3c0c2e9c593bcf0cdd5abae0030a1ddfb36667c9164eedbb7f0a0e9ad223eefe WatchSource:0}: Error finding container 3c0c2e9c593bcf0cdd5abae0030a1ddfb36667c9164eedbb7f0a0e9ad223eefe: Status 404 returned error can't find the container with id 3c0c2e9c593bcf0cdd5abae0030a1ddfb36667c9164eedbb7f0a0e9ad223eefe
Sep 16 00:47:42.714776 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.714741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" event={"ID":"c9bae60a-e806-4940-984d-3b090889a55a","Type":"ContainerStarted","Data":"3c0c2e9c593bcf0cdd5abae0030a1ddfb36667c9164eedbb7f0a0e9ad223eefe"}
Sep 16 00:47:42.716704 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.716673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" event={"ID":"ade09380-f47e-4ac4-a994-74e4992fb3ea","Type":"ContainerStarted","Data":"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"}
Sep 16 00:47:42.716895 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.716858    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:42.717141 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.717107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" event={"ID":"ade09380-f47e-4ac4-a994-74e4992fb3ea","Type":"ContainerStarted","Data":"7c3e7d45a7684d92d7861cd66e9c3842b1eab3fb9c392b6166c721fc272a67a1"}
Sep 16 00:47:42.751176 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:42.751122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" podStartSLOduration=2.325967006 podStartE2EDuration="2.751105439s" totalImagesPullingTime="425.138433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:42.747608221 +0000 UTC m=+832.909118493" watchObservedRunningTime="2026-09-16 00:47:42.751105439 +0000 UTC m=+832.912615708"
Sep 16 00:47:43.723704 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:43.723660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" event={"ID":"c9bae60a-e806-4940-984d-3b090889a55a","Type":"ContainerStarted","Data":"cc8de2803d81ca74e9f519ff31ef7eb2654e29ce4f5b201f10e6c2fe6050d001"}
Sep 16 00:47:43.724092 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:43.723855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:43.745266 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:43.745196    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" podStartSLOduration=2.327023745 podStartE2EDuration="2.74517291s" totalImagesPullingTime="418.149165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:47:43.741519346 +0000 UTC m=+833.903029619" watchObservedRunningTime="2026-09-16 00:47:43.74517291 +0000 UTC m=+833.906683183"
Sep 16 00:47:47.726252 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:47.726213    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:47:48.730170 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:48.730142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:47:49.735300 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:49.735271    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:47:52.713224 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:52.713173    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 00:47:52.713642 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:52.713244    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 00:47:53.796893 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.796852    2577 generic.go:356] "Generic (PLEG): container finished" podID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerID="80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80" exitCode=0
Sep 16 00:47:53.797364 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.796942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerDied","Data":"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"}
Sep 16 00:47:53.797500 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.797482    2577 scope.go:117] "RemoveContainer" containerID="80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"
Sep 16 00:47:53.798660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.798633    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:53.798742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.798700    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:53.798742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:53.798734    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:54.805586 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:54.805543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerStarted","Data":"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"}
Sep 16 00:47:55.274305 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:55.274264    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:47:55.274534 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:55.274515    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 00:47:55.274630 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:55.274566    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 00:47:56.275128 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:56.275089    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 00:47:56.275523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:56.275158    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 00:47:57.275093 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.275016    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 00:47:57.275093 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.275077    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 00:47:57.825551 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.825518    2577 generic.go:356] "Generic (PLEG): container finished" podID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerID="0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d" exitCode=0
Sep 16 00:47:57.825721 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.825589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerDied","Data":"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"}
Sep 16 00:47:57.826070 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.826053    2577 scope.go:117] "RemoveContainer" containerID="0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"
Sep 16 00:47:57.827081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.827055    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:57.827155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.827118    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:57.827155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:57.827146    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:47:58.274752 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:58.274721    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 00:47:58.274982 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:58.274768    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 00:47:58.834583 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:58.834032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerStarted","Data":"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"}
Sep 16 00:47:59.275213 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:59.275174    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 00:47:59.275439 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:59.275242    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 00:47:59.347768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:59.347719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:47:59.348020 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:59.347860    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:47:59.348020 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:47:59.347905    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:00.278381 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:00.278340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:48:00.278762 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:00.278703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:48:00.279345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:00.279327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:48:00.351392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:00.351353    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:48:00.351686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:00.351653    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:01.348143 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:01.348099    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:48:01.348680 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:01.348168    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:02.347650 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:02.347612    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:48:02.349005 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:02.348969    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:03.362418 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:03.362377    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:48:03.363072 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:03.363037    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:04.348028 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:04.347991    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 00:48:04.348212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:04.348048    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 00:48:05.351860 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:05.351829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:48:05.352344 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:05.352146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:48:05.352747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:05.352729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:48:07.349556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.349521    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:48:07.352958 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.352925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.355173 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.355149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4d413f-serving-cert\""
Sep 16 00:48:07.365147 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.365117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:48:07.432520 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.432478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.432686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.432544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.432686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.432568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6km\" (UniqueName: \"kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.432686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.432615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.534458 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.534405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.534644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.534564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.534644 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.534626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6km\" (UniqueName: \"kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.534904 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.534838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.535139 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:48:07.535117    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-serving-cert: secret "maas-api-e2e-inf-b-4d413f-serving-cert" not found
Sep 16 00:48:07.541972 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:48:07.535661    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls podName:8835f67d-d373-45e8-9483-6ea796dcb2b7 nodeName:}" failed. No retries permitted until 2026-09-16 00:48:08.035632661 +0000 UTC m=+858.197142930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls") pod "maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" (UID: "8835f67d-d373-45e8-9483-6ea796dcb2b7") : secret "maas-api-e2e-inf-b-4d413f-serving-cert" not found
Sep 16 00:48:07.541972 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.535854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.541972 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.537686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:07.545994 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:07.545962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6km\" (UniqueName: \"kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:08.039156 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.039123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:08.041650 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.041618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:08.264125 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.264055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:08.329342 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.329312    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:48:08.551917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.551890    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:48:08.553498 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:48:08.553477    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8835f67d_d373_45e8_9483_6ea796dcb2b7.slice/crio-72256d09412e9d0654506062e9c13d8d6cd2d5972ed069ea3a3d92ea5a693214 WatchSource:0}: Error finding container 72256d09412e9d0654506062e9c13d8d6cd2d5972ed069ea3a3d92ea5a693214: Status 404 returned error can't find the container with id 72256d09412e9d0654506062e9c13d8d6cd2d5972ed069ea3a3d92ea5a693214
Sep 16 00:48:08.904326 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:08.904289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" event={"ID":"8835f67d-d373-45e8-9483-6ea796dcb2b7","Type":"ContainerStarted","Data":"72256d09412e9d0654506062e9c13d8d6cd2d5972ed069ea3a3d92ea5a693214"}
Sep 16 00:48:09.910887 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:09.910837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" event={"ID":"8835f67d-d373-45e8-9483-6ea796dcb2b7","Type":"ContainerStarted","Data":"4443991e259d5f50f452da7aed9bc87d0711337e7cc46ae9cce1f059781b1f10"}
Sep 16 00:48:09.911274 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:09.910992    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:09.926772 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:09.926722    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" podStartSLOduration=2.517185729 podStartE2EDuration="2.92669498s" totalImagesPullingTime="409.509251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:48:09.926420327 +0000 UTC m=+860.087930600" watchObservedRunningTime="2026-09-16 00:48:09.92669498 +0000 UTC m=+860.088205254"
Sep 16 00:48:15.921968 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:15.921916    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:48:27.327479 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:27.327446    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:48:50.345390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:50.345311    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:48:50.347153 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:48:50.347127    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:49:12.328259 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:49:12.328230    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:49:40.330502 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:49:40.330472    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:50:28.328174 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:28.328092    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:50:41.756761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:41.756727    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9370-9d5d9a"]
Sep 16 00:50:41.760364 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:41.760344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:41.763148 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:41.763119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qndhv\""
Sep 16 00:50:41.765997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:41.765979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9370-9d5d9a"]
Sep 16 00:50:41.900741 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:41.900711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbjf\" (UniqueName: \"kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf\") pod \"test-curl-9370-9d5d9a\" (UID: \"447051bd-92e4-4f50-ba98-2e38d5f56525\") " pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:42.001617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.001588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbjf\" (UniqueName: \"kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf\") pod \"test-curl-9370-9d5d9a\" (UID: \"447051bd-92e4-4f50-ba98-2e38d5f56525\") " pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:42.009545 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.009485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbjf\" (UniqueName: \"kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf\") pod \"test-curl-9370-9d5d9a\" (UID: \"447051bd-92e4-4f50-ba98-2e38d5f56525\") " pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:42.071568 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.071544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:42.341357 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.341298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9370-9d5d9a"]
Sep 16 00:50:42.344402 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:50:42.344361    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod447051bd_92e4_4f50_ba98_2e38d5f56525.slice/crio-80910a068a2f16fbdfd6f6c3fed006af3a3e04417aa89458eb1cdda0d0c0dd5f WatchSource:0}: Error finding container 80910a068a2f16fbdfd6f6c3fed006af3a3e04417aa89458eb1cdda0d0c0dd5f: Status 404 returned error can't find the container with id 80910a068a2f16fbdfd6f6c3fed006af3a3e04417aa89458eb1cdda0d0c0dd5f
Sep 16 00:50:42.349684 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.349660    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:50:42.736849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:42.736767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9370-9d5d9a" event={"ID":"447051bd-92e4-4f50-ba98-2e38d5f56525","Type":"ContainerStarted","Data":"80910a068a2f16fbdfd6f6c3fed006af3a3e04417aa89458eb1cdda0d0c0dd5f"}
Sep 16 00:50:44.750672 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:44.750590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9370-9d5d9a" event={"ID":"447051bd-92e4-4f50-ba98-2e38d5f56525","Type":"ContainerStarted","Data":"1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd"}
Sep 16 00:50:44.765934 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:44.765889    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9370-9d5d9a" podStartSLOduration=1.75305822 podStartE2EDuration="3.765872214s" totalImagesPullingTime="2.012813994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:50:44.763540976 +0000 UTC m=+1014.925051250" watchObservedRunningTime="2026-09-16 00:50:44.765872214 +0000 UTC m=+1014.927382487"
Sep 16 00:50:45.389761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.389726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9370-9d5d9a"]
Sep 16 00:50:45.393924 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.393900    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9370-9d5d9a"]
Sep 16 00:50:45.527827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.527791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9370-bb1b9a"]
Sep 16 00:50:45.528368 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.528349    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" containerName="test-curl-9370-9d5d9a"
Sep 16 00:50:45.528481 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.528370    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" containerName="test-curl-9370-9d5d9a"
Sep 16 00:50:45.528537 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.528522    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" containerName="test-curl-9370-9d5d9a"
Sep 16 00:50:45.531909 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.531884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:45.535281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.535257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9370-bb1b9a"]
Sep 16 00:50:45.636119 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.636081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkjv\" (UniqueName: \"kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv\") pod \"test-curl-9370-bb1b9a\" (UID: \"161f8499-491e-46ce-b39c-e574feba1ad9\") " pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:45.737625 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.737596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkjv\" (UniqueName: \"kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv\") pod \"test-curl-9370-bb1b9a\" (UID: \"161f8499-491e-46ce-b39c-e574feba1ad9\") " pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:45.745966 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.745924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkjv\" (UniqueName: \"kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv\") pod \"test-curl-9370-bb1b9a\" (UID: \"161f8499-491e-46ce-b39c-e574feba1ad9\") " pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:45.756524 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:45.756503    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9370-9d5d9a" container="test-curl-9370-9d5d9a"
Sep 16 00:50:45.845533 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:45.845441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:46.115916 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:46.115888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9370-bb1b9a"]
Sep 16 00:50:46.117396 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:50:46.117374    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161f8499_491e_46ce_b39c_e574feba1ad9.slice/crio-1ea05e9168052a060e9b1e0063d070d67badd905ff33e776fb7609f34d1d11b0 WatchSource:0}: Error finding container 1ea05e9168052a060e9b1e0063d070d67badd905ff33e776fb7609f34d1d11b0: Status 404 returned error can't find the container with id 1ea05e9168052a060e9b1e0063d070d67badd905ff33e776fb7609f34d1d11b0
Sep 16 00:50:46.763466 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:46.763426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9370-bb1b9a" event={"ID":"161f8499-491e-46ce-b39c-e574feba1ad9","Type":"ContainerStarted","Data":"48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644"}
Sep 16 00:50:46.763920 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:46.763475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9370-bb1b9a" event={"ID":"161f8499-491e-46ce-b39c-e574feba1ad9","Type":"ContainerStarted","Data":"1ea05e9168052a060e9b1e0063d070d67badd905ff33e776fb7609f34d1d11b0"}
Sep 16 00:50:46.763920 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:46.763522    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9370-9d5d9a" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" containerName="test-curl-9370-9d5d9a" containerID="cri-o://1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd" gracePeriod=2
Sep 16 00:50:46.776660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:46.776513    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9370-bb1b9a" podStartSLOduration=1.776494826 podStartE2EDuration="1.776494826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:50:46.775803928 +0000 UTC m=+1016.937314200" watchObservedRunningTime="2026-09-16 00:50:46.776494826 +0000 UTC m=+1016.938005100"
Sep 16 00:50:47.407440 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:47.407408    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9370-bb1b9a"]
Sep 16 00:50:47.411592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:47.411570    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9370-bb1b9a"]
Sep 16 00:50:47.769187 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:47.769152    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9370-bb1b9a" container="test-curl-9370-bb1b9a"
Sep 16 00:50:48.775311 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:48.775277    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9370-bb1b9a" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" containerName="test-curl-9370-bb1b9a" containerID="cri-o://48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644" gracePeriod=2
Sep 16 00:50:48.966649 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:48.966623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:48.970921 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:48.970902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcbjf\" (UniqueName: \"kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf\") pod \"447051bd-92e4-4f50-ba98-2e38d5f56525\" (UID: \"447051bd-92e4-4f50-ba98-2e38d5f56525\") "
Sep 16 00:50:48.972917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:48.972894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf" (OuterVolumeSpecName: "kube-api-access-hcbjf") pod "447051bd-92e4-4f50-ba98-2e38d5f56525" (UID: "447051bd-92e4-4f50-ba98-2e38d5f56525"). InnerVolumeSpecName "kube-api-access-hcbjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:50:49.071857 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.071781    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcbjf\" (UniqueName: \"kubernetes.io/projected/447051bd-92e4-4f50-ba98-2e38d5f56525-kube-api-access-hcbjf\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:50:49.781838 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.781796    2577 generic.go:356] "Generic (PLEG): container finished" podID="447051bd-92e4-4f50-ba98-2e38d5f56525" containerID="1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd" exitCode=137
Sep 16 00:50:49.781838 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.781838    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:49.782394 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.781905    2577 scope.go:117] "RemoveContainer" containerID="1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd"
Sep 16 00:50:49.791722 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.791699    2577 scope.go:117] "RemoveContainer" containerID="1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd"
Sep 16 00:50:49.792058 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:49.792035    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd\": container with ID starting with 1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd not found: ID does not exist" containerID="1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd"
Sep 16 00:50:49.792118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:49.792070    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd"} err="failed to get container status \"1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd\": rpc error: code = NotFound desc = could not find container \"1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd\": container with ID starting with 1e2ba49421249ae2f7a5b5ab3234bc0cf591ce5d7e20f6adbee08e3f86b87ddd not found: ID does not exist"
Sep 16 00:50:49.792624 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:49.792598    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9370-9d5d9a\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:50.332128 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:50.332084    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9370-9d5d9a\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" pod="opendatahub/test-curl-9370-9d5d9a"
Sep 16 00:50:50.334826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:50.334795    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447051bd-92e4-4f50-ba98-2e38d5f56525" path="/var/lib/kubelet/pods/447051bd-92e4-4f50-ba98-2e38d5f56525/volumes"
Sep 16 00:50:50.974486 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:50.974459    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:50.986618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:50.986596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkjv\" (UniqueName: \"kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv\") pod \"161f8499-491e-46ce-b39c-e574feba1ad9\" (UID: \"161f8499-491e-46ce-b39c-e574feba1ad9\") "
Sep 16 00:50:50.988771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:50.988749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv" (OuterVolumeSpecName: "kube-api-access-9qkjv") pod "161f8499-491e-46ce-b39c-e574feba1ad9" (UID: "161f8499-491e-46ce-b39c-e574feba1ad9"). InnerVolumeSpecName "kube-api-access-9qkjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:50:51.087526 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.087438    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkjv\" (UniqueName: \"kubernetes.io/projected/161f8499-491e-46ce-b39c-e574feba1ad9-kube-api-access-9qkjv\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:50:51.801842 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.801807    2577 generic.go:356] "Generic (PLEG): container finished" podID="161f8499-491e-46ce-b39c-e574feba1ad9" containerID="48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644" exitCode=137
Sep 16 00:50:51.802064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.801854    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:51.802064 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.801858    2577 scope.go:117] "RemoveContainer" containerID="48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644"
Sep 16 00:50:51.811277 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.811259    2577 scope.go:117] "RemoveContainer" containerID="48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644"
Sep 16 00:50:51.811552 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:51.811532    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644\": container with ID starting with 48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644 not found: ID does not exist" containerID="48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644"
Sep 16 00:50:51.811624 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:51.811560    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644"} err="failed to get container status \"48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644\": rpc error: code = NotFound desc = could not find container \"48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644\": container with ID starting with 48fa26b684aff845b8229a80a8d09adfbd9f7a5b376d1e38c714556a4b14c644 not found: ID does not exist"
Sep 16 00:50:51.812174 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:50:51.812156    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9370-bb1b9a\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" pod="opendatahub/test-curl-9370-bb1b9a"
Sep 16 00:50:52.334181 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:52.334152    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" path="/var/lib/kubelet/pods/161f8499-491e-46ce-b39c-e574feba1ad9/volumes"
Sep 16 00:50:58.328150 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:50:58.328112    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:51:05.406496 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.406465    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:51:05.406974 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.406943    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" containerName="test-curl-9370-bb1b9a"
Sep 16 00:51:05.407048 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.406977    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" containerName="test-curl-9370-bb1b9a"
Sep 16 00:51:05.407083 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.407065    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="161f8499-491e-46ce-b39c-e574feba1ad9" containerName="test-curl-9370-bb1b9a"
Sep 16 00:51:05.418768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.418739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.421392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.421367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:51:05.422537 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.422512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"distinct-2-w0-5270e2-kserve-self-signed-certs\""
Sep 16 00:51:05.422651 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.422548    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"kube-root-ca.crt\""
Sep 16 00:51:05.422823 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.422806    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"openshift-service-ca.crt\""
Sep 16 00:51:05.423055 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.423027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5270e2\"/\"default-dockercfg-fmb5k\""
Sep 16 00:51:05.528332 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.528332 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.528595 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.528595 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.528595 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sddb\" (UniqueName: \"kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.528595 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.528535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629356 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629526 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629526 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629526 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629706 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629706 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sddb\" (UniqueName: \"kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.629870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.629822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.631990 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.631966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.632177 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.632157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.636990 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.636965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sddb\" (UniqueName: \"kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb\") pod \"distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") " pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:05.730245 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:05.730164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:06.003982 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:06.003941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:51:06.006258 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:51:06.006230    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1520e620_b82e_4b1c_8e62_692f64a790ec.slice/crio-64d1bac88cfcc7bbce8d8e3453c3647e3a4e981b6fdb73f82074c3051d61c790 WatchSource:0}: Error finding container 64d1bac88cfcc7bbce8d8e3453c3647e3a4e981b6fdb73f82074c3051d61c790: Status 404 returned error can't find the container with id 64d1bac88cfcc7bbce8d8e3453c3647e3a4e981b6fdb73f82074c3051d61c790
Sep 16 00:51:06.889582 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:06.889542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" event={"ID":"1520e620-b82e-4b1c-8e62-692f64a790ec","Type":"ContainerStarted","Data":"18652dd5d4e08357d0e6b98e421c3effe89259e21013a6366317f1c917b23fe9"}
Sep 16 00:51:06.889988 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:06.889590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" event={"ID":"1520e620-b82e-4b1c-8e62-692f64a790ec","Type":"ContainerStarted","Data":"64d1bac88cfcc7bbce8d8e3453c3647e3a4e981b6fdb73f82074c3051d61c790"}
Sep 16 00:51:06.905938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:06.905892    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" podStartSLOduration=1.905875897 podStartE2EDuration="1.905875897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:06.904219801 +0000 UTC m=+1037.065730074" watchObservedRunningTime="2026-09-16 00:51:06.905875897 +0000 UTC m=+1037.067386168"
Sep 16 00:51:15.730580 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:15.730544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:15.731127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:15.730595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:15.743097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:15.743074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:15.952329 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:15.952299    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:51:18.846456 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:18.846423    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:51:18.846929 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:18.846700    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" podUID="a8330023-ec14-4c23-ba7c-fea1d220636b" containerName="maas-api" containerID="cri-o://3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8" gracePeriod=30
Sep 16 00:51:19.316762 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.316739    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:51:19.472017 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.471927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs\") pod \"a8330023-ec14-4c23-ba7c-fea1d220636b\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") "
Sep 16 00:51:19.472165 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.472073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb7cj\" (UniqueName: \"kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj\") pod \"a8330023-ec14-4c23-ba7c-fea1d220636b\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") "
Sep 16 00:51:19.472165 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.472128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") pod \"a8330023-ec14-4c23-ba7c-fea1d220636b\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") "
Sep 16 00:51:19.472165 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.472160    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca\") pod \"a8330023-ec14-4c23-ba7c-fea1d220636b\" (UID: \"a8330023-ec14-4c23-ba7c-fea1d220636b\") "
Sep 16 00:51:19.472912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.472566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8330023-ec14-4c23-ba7c-fea1d220636b" (UID: "a8330023-ec14-4c23-ba7c-fea1d220636b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:19.472912 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.472701    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8330023-ec14-4c23-ba7c-fea1d220636b-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:19.474539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.474499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8330023-ec14-4c23-ba7c-fea1d220636b" (UID: "a8330023-ec14-4c23-ba7c-fea1d220636b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:19.474539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.474518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8330023-ec14-4c23-ba7c-fea1d220636b" (UID: "a8330023-ec14-4c23-ba7c-fea1d220636b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:19.474539 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.474524    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj" (OuterVolumeSpecName: "kube-api-access-zb7cj") pod "a8330023-ec14-4c23-ba7c-fea1d220636b" (UID: "a8330023-ec14-4c23-ba7c-fea1d220636b"). InnerVolumeSpecName "kube-api-access-zb7cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:19.573708 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.573672    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:19.573708 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.573700    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8330023-ec14-4c23-ba7c-fea1d220636b-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:19.573708 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.573712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb7cj\" (UniqueName: \"kubernetes.io/projected/a8330023-ec14-4c23-ba7c-fea1d220636b-kube-api-access-zb7cj\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:19.963313 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.963278    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8330023-ec14-4c23-ba7c-fea1d220636b" containerID="3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8" exitCode=0
Sep 16 00:51:19.963793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.963329    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"
Sep 16 00:51:19.963793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.963365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" event={"ID":"a8330023-ec14-4c23-ba7c-fea1d220636b","Type":"ContainerDied","Data":"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"}
Sep 16 00:51:19.963793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.963421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9" event={"ID":"a8330023-ec14-4c23-ba7c-fea1d220636b","Type":"ContainerDied","Data":"6dc8afef4d71b540574bd325a21cf9962af3e971fe7b216bd34e9e3ba45ab7eb"}
Sep 16 00:51:19.963793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.963444    2577 scope.go:117] "RemoveContainer" containerID="3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"
Sep 16 00:51:19.974469 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.974453    2577 scope.go:117] "RemoveContainer" containerID="3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"
Sep 16 00:51:19.974731 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:51:19.974713    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8\": container with ID starting with 3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8 not found: ID does not exist" containerID="3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"
Sep 16 00:51:19.974787 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.974739    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8"} err="failed to get container status \"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8\": rpc error: code = NotFound desc = could not find container \"3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8\": container with ID starting with 3b3ec61a0f95a9f5420243d8d657e6d12f80300169da62387f3d78a210efa5a8 not found: ID does not exist"
Sep 16 00:51:19.989103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.989079    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:51:19.993218 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:19.993191    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ecb7-5645884fb-cbdb9"]
Sep 16 00:51:20.335937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:20.335903    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8330023-ec14-4c23-ba7c-fea1d220636b" path="/var/lib/kubelet/pods/a8330023-ec14-4c23-ba7c-fea1d220636b/volumes"
Sep 16 00:51:25.612563 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:25.612525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:51:25.613103 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:25.612797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" containerID="cri-o://edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53" gracePeriod=30
Sep 16 00:51:30.281066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.281031    2577 patch_prober.go:28] interesting pod/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 00:51:30.281474 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.281095    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:51:30.886618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.886560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:51:30.987372 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987372 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987515    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987539    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.987626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vvr5\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5\") pod \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\" (UID: \"599cb85a-cb65-4bd6-b651-6abc8e49dcc1\") "
Sep 16 00:51:30.988026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:30.988026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:30.988026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:30.988026 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:30.988202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.988064    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:30.988202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.988083    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:30.988202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.988098    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:30.988202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.987838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:30.990014 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.989990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5" (OuterVolumeSpecName: "kube-api-access-7vvr5") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "kube-api-access-7vvr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:30.990014 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.989990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:51:30.990341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.990321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 00:51:30.990422 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:30.990407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "599cb85a-cb65-4bd6-b651-6abc8e49dcc1" (UID: "599cb85a-cb65-4bd6-b651-6abc8e49dcc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:31.030928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.030898    2577 generic.go:356] "Generic (PLEG): container finished" podID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerID="edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53" exitCode=0
Sep 16 00:51:31.031096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.030980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"
Sep 16 00:51:31.031096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.030990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerDied","Data":"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"}
Sep 16 00:51:31.031096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.031048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8" event={"ID":"599cb85a-cb65-4bd6-b651-6abc8e49dcc1","Type":"ContainerDied","Data":"345ba2094d9d02eb0f682b47a0b5ab0218cbad72a867c258a476e5f7c311909f"}
Sep 16 00:51:31.031096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.031076    2577 scope.go:117] "RemoveContainer" containerID="edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"
Sep 16 00:51:31.047919 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.047898    2577 scope.go:117] "RemoveContainer" containerID="80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"
Sep 16 00:51:31.056553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.056535    2577 scope.go:117] "RemoveContainer" containerID="edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"
Sep 16 00:51:31.056843 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:51:31.056824    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53\": container with ID starting with edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53 not found: ID does not exist" containerID="edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"
Sep 16 00:51:31.056900 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.056850    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53"} err="failed to get container status \"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53\": rpc error: code = NotFound desc = could not find container \"edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53\": container with ID starting with edc448b3d776a494f09244844852ab4b1babd78c7435775c05b4e6e80625fd53 not found: ID does not exist"
Sep 16 00:51:31.056900 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.056868    2577 scope.go:117] "RemoveContainer" containerID="80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"
Sep 16 00:51:31.057153 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:51:31.057134    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80\": container with ID starting with 80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80 not found: ID does not exist" containerID="80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"
Sep 16 00:51:31.057212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.057160    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80"} err="failed to get container status \"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80\": rpc error: code = NotFound desc = could not find container \"80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80\": container with ID starting with 80a5a31b0b349e5b477c6c5e51e731e408d50cd944e5382653cd9702541c1f80 not found: ID does not exist"
Sep 16 00:51:31.058706 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.058683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:51:31.061464 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.061444    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-66ecb7-openshift-default-76c7b559d4-t8kz8"]
Sep 16 00:51:31.089289 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089264    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:31.089289 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089289    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:31.089419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089299    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:31.089419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089307    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:31.089419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089315    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:31.089419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:31.089323    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvr5\" (UniqueName: \"kubernetes.io/projected/599cb85a-cb65-4bd6-b651-6abc8e49dcc1-kube-api-access-7vvr5\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:32.334437 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:32.334409    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" path="/var/lib/kubelet/pods/599cb85a-cb65-4bd6-b651-6abc8e49dcc1/volumes"
Sep 16 00:51:36.837691 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:36.837651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:51:36.838167 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:36.837905    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" podUID="8835f67d-d373-45e8-9483-6ea796dcb2b7" containerName="maas-api" containerID="cri-o://4443991e259d5f50f452da7aed9bc87d0711337e7cc46ae9cce1f059781b1f10" gracePeriod=30
Sep 16 00:51:37.067315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.067285    2577 generic.go:356] "Generic (PLEG): container finished" podID="8835f67d-d373-45e8-9483-6ea796dcb2b7" containerID="4443991e259d5f50f452da7aed9bc87d0711337e7cc46ae9cce1f059781b1f10" exitCode=0
Sep 16 00:51:37.067486 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.067364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" event={"ID":"8835f67d-d373-45e8-9483-6ea796dcb2b7","Type":"ContainerDied","Data":"4443991e259d5f50f452da7aed9bc87d0711337e7cc46ae9cce1f059781b1f10"}
Sep 16 00:51:37.107976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.107924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:51:37.144915 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.144879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca\") pod \"8835f67d-d373-45e8-9483-6ea796dcb2b7\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") "
Sep 16 00:51:37.145118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.145034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") pod \"8835f67d-d373-45e8-9483-6ea796dcb2b7\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") "
Sep 16 00:51:37.145118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.145072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s6km\" (UniqueName: \"kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km\") pod \"8835f67d-d373-45e8-9483-6ea796dcb2b7\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") "
Sep 16 00:51:37.145118 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.145103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs\") pod \"8835f67d-d373-45e8-9483-6ea796dcb2b7\" (UID: \"8835f67d-d373-45e8-9483-6ea796dcb2b7\") "
Sep 16 00:51:37.145304 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.145230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8835f67d-d373-45e8-9483-6ea796dcb2b7" (UID: "8835f67d-d373-45e8-9483-6ea796dcb2b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:51:37.145439 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.145419    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8835f67d-d373-45e8-9483-6ea796dcb2b7-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:37.147514 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.147485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km" (OuterVolumeSpecName: "kube-api-access-6s6km") pod "8835f67d-d373-45e8-9483-6ea796dcb2b7" (UID: "8835f67d-d373-45e8-9483-6ea796dcb2b7"). InnerVolumeSpecName "kube-api-access-6s6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:51:37.147873 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.147851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8835f67d-d373-45e8-9483-6ea796dcb2b7" (UID: "8835f67d-d373-45e8-9483-6ea796dcb2b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:37.147994 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.147965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8835f67d-d373-45e8-9483-6ea796dcb2b7" (UID: "8835f67d-d373-45e8-9483-6ea796dcb2b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:51:37.246386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.246347    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:37.246386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.246388    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s6km\" (UniqueName: \"kubernetes.io/projected/8835f67d-d373-45e8-9483-6ea796dcb2b7-kube-api-access-6s6km\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:37.246581 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:37.246404    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8835f67d-d373-45e8-9483-6ea796dcb2b7-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:51:38.074130 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.074088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc" event={"ID":"8835f67d-d373-45e8-9483-6ea796dcb2b7","Type":"ContainerDied","Data":"72256d09412e9d0654506062e9c13d8d6cd2d5972ed069ea3a3d92ea5a693214"}
Sep 16 00:51:38.074605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.074555    2577 scope.go:117] "RemoveContainer" containerID="4443991e259d5f50f452da7aed9bc87d0711337e7cc46ae9cce1f059781b1f10"
Sep 16 00:51:38.074673 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.074583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"
Sep 16 00:51:38.100448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.100425    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:51:38.104343 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.104321    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d413f-5bcfd5c798-gtpgc"]
Sep 16 00:51:38.334518 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:38.334439    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8835f67d-d373-45e8-9483-6ea796dcb2b7" path="/var/lib/kubelet/pods/8835f67d-d373-45e8-9483-6ea796dcb2b7/volumes"
Sep 16 00:51:42.836440 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.836360    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:51:42.836991 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.836972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:42.837046 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.836996    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:42.837046 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837013    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8330023-ec14-4c23-ba7c-fea1d220636b" containerName="maas-api"
Sep 16 00:51:42.837046 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837024    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8330023-ec14-4c23-ba7c-fea1d220636b" containerName="maas-api"
Sep 16 00:51:42.837141 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837059    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8835f67d-d373-45e8-9483-6ea796dcb2b7" containerName="maas-api"
Sep 16 00:51:42.837141 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837068    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8835f67d-d373-45e8-9483-6ea796dcb2b7" containerName="maas-api"
Sep 16 00:51:42.837204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837190    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:42.837237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837202    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8330023-ec14-4c23-ba7c-fea1d220636b" containerName="maas-api"
Sep 16 00:51:42.837237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8835f67d-d373-45e8-9483-6ea796dcb2b7" containerName="maas-api"
Sep 16 00:51:42.837237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.837220    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:42.842157 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.842132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.844696 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.844674    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-629e17ec-openshift-default-dockercfg-znw4f\""
Sep 16 00:51:42.852617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.852590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:51:42.897455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwwr\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897628 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897628 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897628 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897776 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897776 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897776 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897776 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.897914 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.897780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwwr\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999836 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999836 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:42.999836 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.000004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.000004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:42.999945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.000315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.000294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.001916 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.001892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.002296 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.002279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.007352 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.007330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.007490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.007472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwwr\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr\") pod \"e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") " pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.155112 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.155020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:43.439008 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.438977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:51:43.440516 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:51:43.440490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549a874d_ea31_45ea_8358_25f8c0dbd547.slice/crio-ceab478d7783838a479f2135714831b1d4c4977d702deec65755f283b91a7082 WatchSource:0}: Error finding container ceab478d7783838a479f2135714831b1d4c4977d702deec65755f283b91a7082: Status 404 returned error can't find the container with id ceab478d7783838a479f2135714831b1d4c4977d702deec65755f283b91a7082
Sep 16 00:51:43.448299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.448245    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:43.448417 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.448384    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:43.448523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:43.448474    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:44.108868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.108829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerStarted","Data":"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"}
Sep 16 00:51:44.109268 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.108921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerStarted","Data":"ceab478d7783838a479f2135714831b1d4c4977d702deec65755f283b91a7082"}
Sep 16 00:51:44.127725 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.127673    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" podStartSLOduration=2.127654402 podStartE2EDuration="2.127654402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:44.127275851 +0000 UTC m=+1074.288786136" watchObservedRunningTime="2026-09-16 00:51:44.127654402 +0000 UTC m=+1074.289164675"
Sep 16 00:51:44.155392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.155271    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:44.160483 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.160462    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:44.769869 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.769835    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 00:51:44.770634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.770612    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:44.770634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.770634    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="599cb85a-cb65-4bd6-b651-6abc8e49dcc1" containerName="istio-proxy"
Sep 16 00:51:44.774619 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.774593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.777675 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.777651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-108960-openshift-default-dockercfg-m78g9\""
Sep 16 00:51:44.790756 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.790613    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 00:51:44.816497 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816643 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816643 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816643 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndsm\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.816825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.817054 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.816829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918330 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918544 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndsm\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.918984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.918611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.919123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.919094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.919226 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.919196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.919404 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.919368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.919590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.919538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.919893 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.919870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.921685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.921655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.922762 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.922730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.927317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.927244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:44.927634 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:44.927592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndsm\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm\") pod \"e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") " pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:45.087729 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.087643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:45.114794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.114764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:45.115593 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.115569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:51:45.376496 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.376466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 00:51:45.377936 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:51:45.377905    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode688e66c_2e95_4b5d_8448_f1e0888efaa7.slice/crio-03888b5227815826d1e4cded05361ae18574b54a869574ce81e5b5fb4ff6c31f WatchSource:0}: Error finding container 03888b5227815826d1e4cded05361ae18574b54a869574ce81e5b5fb4ff6c31f: Status 404 returned error can't find the container with id 03888b5227815826d1e4cded05361ae18574b54a869574ce81e5b5fb4ff6c31f
Sep 16 00:51:45.380290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.380260    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:45.380367 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.380325    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:45.380367 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:45.380353    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:51:46.122019 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:46.121982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerStarted","Data":"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"}
Sep 16 00:51:46.122513 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:46.122046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerStarted","Data":"03888b5227815826d1e4cded05361ae18574b54a869574ce81e5b5fb4ff6c31f"}
Sep 16 00:51:46.146919 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:46.146861    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podStartSLOduration=2.14684729 podStartE2EDuration="2.14684729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:46.144109101 +0000 UTC m=+1076.305619374" watchObservedRunningTime="2026-09-16 00:51:46.14684729 +0000 UTC m=+1076.308357561"
Sep 16 00:51:47.088579 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.088544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:47.093830 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.093805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:47.128865 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.128826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:47.129608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.129584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:51:47.520144 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.520109    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:51:47.527208 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.527183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.530824 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.530800    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:51:47.532415 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.532386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-629e17ec-serving-cert\""
Sep 16 00:51:47.644361 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.644327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.644523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.644381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs54\" (UniqueName: \"kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.644523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.644434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.644621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.644522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.745604 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.745566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.745771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.745648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.745771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.745694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs54\" (UniqueName: \"kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.745771 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.745738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.746354 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.746321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.748845 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.748822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.748932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.748890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.756301 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.756278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs54\" (UniqueName: \"kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54\") pod \"maas-api-e2e-ait-629e17ec-7b7975d7b-d7675\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:47.843623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:47.843549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:48.123566 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:48.123538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:51:48.125599 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:51:48.125567    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c049e4_0dd3_429d_b257_f319edb696e0.slice/crio-3f2222992268dbf9c2a01103caa777f13bb0182fd9f6b972e49dc856d9c0b949 WatchSource:0}: Error finding container 3f2222992268dbf9c2a01103caa777f13bb0182fd9f6b972e49dc856d9c0b949: Status 404 returned error can't find the container with id 3f2222992268dbf9c2a01103caa777f13bb0182fd9f6b972e49dc856d9c0b949
Sep 16 00:51:48.139074 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:48.138993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" event={"ID":"30c049e4-0dd3-429d-b257-f319edb696e0","Type":"ContainerStarted","Data":"3f2222992268dbf9c2a01103caa777f13bb0182fd9f6b972e49dc856d9c0b949"}
Sep 16 00:51:49.147247 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:49.147206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" event={"ID":"30c049e4-0dd3-429d-b257-f319edb696e0","Type":"ContainerStarted","Data":"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"}
Sep 16 00:51:49.147670 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:49.147482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:49.169244 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:49.169198    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" podStartSLOduration=1.778019264 podStartE2EDuration="2.169183056s" totalImagesPullingTime="391.163792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:49.168599387 +0000 UTC m=+1079.330109658" watchObservedRunningTime="2026-09-16 00:51:49.169183056 +0000 UTC m=+1079.330693327"
Sep 16 00:51:51.293779 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.293741    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 00:51:51.299367 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.299342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.301711 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.301692    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-108960-serving-cert\""
Sep 16 00:51:51.312537 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.312515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 00:51:51.380617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.380588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.380617 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.380622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.380848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.380727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.380848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.380793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scvc\" (UniqueName: \"kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.481825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.481789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.481825 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.481821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.482102 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.481868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.482102 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.481907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scvc\" (UniqueName: \"kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.482102 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:51:51.482016    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-serving-cert: secret "maas-api-e2e-shared-a-w2-108960-serving-cert" not found
Sep 16 00:51:51.482102 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:51:51.482087    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls podName:6c258f94-0cb9-4a70-8981-d1079ccc135b nodeName:}" failed. No retries permitted until 2026-09-16 00:51:51.98206659 +0000 UTC m=+1082.143576841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" (UID: "6c258f94-0cb9-4a70-8981-d1079ccc135b") : secret "maas-api-e2e-shared-a-w2-108960-serving-cert" not found
Sep 16 00:51:51.482659 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.482640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.484621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.484593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.490249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.490223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scvc\" (UniqueName: \"kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.987189 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.987157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:51.990016 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:51.989983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:52.213391 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:52.213359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:52.488830 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:52.488787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 00:51:52.490466 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:51:52.490439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c258f94_0cb9_4a70_8981_d1079ccc135b.slice/crio-0ea258297228d7ff06dab25ebdc9a7253c93ab42b2bf663e22f4a585321edaea WatchSource:0}: Error finding container 0ea258297228d7ff06dab25ebdc9a7253c93ab42b2bf663e22f4a585321edaea: Status 404 returned error can't find the container with id 0ea258297228d7ff06dab25ebdc9a7253c93ab42b2bf663e22f4a585321edaea
Sep 16 00:51:53.174250 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:53.174156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" event={"ID":"6c258f94-0cb9-4a70-8981-d1079ccc135b","Type":"ContainerStarted","Data":"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"}
Sep 16 00:51:53.174580 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:53.174263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" event={"ID":"6c258f94-0cb9-4a70-8981-d1079ccc135b","Type":"ContainerStarted","Data":"0ea258297228d7ff06dab25ebdc9a7253c93ab42b2bf663e22f4a585321edaea"}
Sep 16 00:51:53.174580 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:53.174421    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:51:53.192367 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:53.192317    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" podStartSLOduration=1.848525021 podStartE2EDuration="2.192304086s" totalImagesPullingTime="343.779065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:51:53.189602448 +0000 UTC m=+1083.351112719" watchObservedRunningTime="2026-09-16 00:51:53.192304086 +0000 UTC m=+1083.353814357"
Sep 16 00:51:55.157928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:55.157895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:51:57.328314 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:57.328284    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:51:59.185705 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:51:59.185674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 00:52:13.328543 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:13.328501    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:52:15.115904 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.115879    2577 patch_prober.go:28] interesting pod/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 00:52:15.116281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.115933    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 00:52:15.312943 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.312910    2577 generic.go:356] "Generic (PLEG): container finished" podID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerID="eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4" exitCode=0
Sep 16 00:52:15.313137 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.312983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerDied","Data":"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"}
Sep 16 00:52:15.313495 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.313475    2577 scope.go:117] "RemoveContainer" containerID="eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"
Sep 16 00:52:15.314763 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.314735    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:15.314848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.314760    2577 generic.go:356] "Generic (PLEG): container finished" podID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerID="62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc" exitCode=0
Sep 16 00:52:15.314848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.314807    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:15.314848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.314823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerDied","Data":"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"}
Sep 16 00:52:15.314848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.314838    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:15.315176 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.315162    2577 scope.go:117] "RemoveContainer" containerID="62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"
Sep 16 00:52:15.316152 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.316125    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:15.316255 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.316190    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:15.316255 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:15.316234    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:52:16.321943 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:16.321908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerStarted","Data":"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"}
Sep 16 00:52:16.323970 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:16.323932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerStarted","Data":"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"}
Sep 16 00:52:17.089376 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.089341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:52:17.089768 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.089746    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 16 00:52:17.089916 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.089895    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 16 00:52:17.129630 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.129598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:52:17.155835 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.155803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:52:17.156503 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.156447    2577 patch_prober.go:28] interesting pod/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 00:52:17.157803 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.156747    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 00:52:17.615299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.615259    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:52:17.615739 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:17.615500    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" podUID="30c049e4-0dd3-429d-b257-f319edb696e0" containerName="maas-api" containerID="cri-o://2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23" gracePeriod=30
Sep 16 00:52:18.038505 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.038474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:52:18.112585 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.095001    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 16 00:52:18.112585 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.112041    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 16 00:52:18.112585 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.111006    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:52:18.159977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.156312    2577 patch_prober.go:28] interesting pod/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 16 00:52:18.159977 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.156374    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 16 00:52:18.201976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.201323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca\") pod \"30c049e4-0dd3-429d-b257-f319edb696e0\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") "
Sep 16 00:52:18.201976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.201412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qs54\" (UniqueName: \"kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54\") pod \"30c049e4-0dd3-429d-b257-f319edb696e0\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") "
Sep 16 00:52:18.201976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.201445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs\") pod \"30c049e4-0dd3-429d-b257-f319edb696e0\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") "
Sep 16 00:52:18.201976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.201505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls\") pod \"30c049e4-0dd3-429d-b257-f319edb696e0\" (UID: \"30c049e4-0dd3-429d-b257-f319edb696e0\") "
Sep 16 00:52:18.204618 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.203386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30c049e4-0dd3-429d-b257-f319edb696e0" (UID: "30c049e4-0dd3-429d-b257-f319edb696e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:52:18.205612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.205178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54" (OuterVolumeSpecName: "kube-api-access-5qs54") pod "30c049e4-0dd3-429d-b257-f319edb696e0" (UID: "30c049e4-0dd3-429d-b257-f319edb696e0"). InnerVolumeSpecName "kube-api-access-5qs54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:52:18.205612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.205220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "30c049e4-0dd3-429d-b257-f319edb696e0" (UID: "30c049e4-0dd3-429d-b257-f319edb696e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:52:18.205612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.205537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30c049e4-0dd3-429d-b257-f319edb696e0" (UID: "30c049e4-0dd3-429d-b257-f319edb696e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:52:18.302744 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.302655    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qs54\" (UniqueName: \"kubernetes.io/projected/30c049e4-0dd3-429d-b257-f319edb696e0-kube-api-access-5qs54\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:18.302744 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.302687    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:18.302744 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.302698    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c049e4-0dd3-429d-b257-f319edb696e0-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:18.302744 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.302707    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c049e4-0dd3-429d-b257-f319edb696e0-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:18.335957 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.335912    2577 generic.go:356] "Generic (PLEG): container finished" podID="30c049e4-0dd3-429d-b257-f319edb696e0" containerID="2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23" exitCode=0
Sep 16 00:52:18.336097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.335988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" event={"ID":"30c049e4-0dd3-429d-b257-f319edb696e0","Type":"ContainerDied","Data":"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"}
Sep 16 00:52:18.336097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.336009    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"
Sep 16 00:52:18.336097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.336032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675" event={"ID":"30c049e4-0dd3-429d-b257-f319edb696e0","Type":"ContainerDied","Data":"3f2222992268dbf9c2a01103caa777f13bb0182fd9f6b972e49dc856d9c0b949"}
Sep 16 00:52:18.336097 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.336055    2577 scope.go:117] "RemoveContainer" containerID="2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"
Sep 16 00:52:18.336322 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.336299    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy" containerID="cri-o://8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed" gracePeriod=30
Sep 16 00:52:18.382108 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.378090    2577 scope.go:117] "RemoveContainer" containerID="2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"
Sep 16 00:52:18.384395 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:18.384345    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23\": container with ID starting with 2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23 not found: ID does not exist" containerID="2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"
Sep 16 00:52:18.384533 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.384406    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23"} err="failed to get container status \"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23\": rpc error: code = NotFound desc = could not find container \"2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23\": container with ID starting with 2befaf945c424485bef4c07784ecc645853f3d5b75ed6b2c6abae89347135f23 not found: ID does not exist"
Sep 16 00:52:18.395774 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.395749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:52:18.399710 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:18.399688    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-629e17ec-7b7975d7b-d7675"]
Sep 16 00:52:19.089340 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:19.088858    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 16 00:52:19.089340 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:19.088938    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 16 00:52:20.088992 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:20.088926    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 16 00:52:20.089206 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:20.089020    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 16 00:52:20.333754 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:20.333722    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c049e4-0dd3-429d-b257-f319edb696e0" path="/var/lib/kubelet/pods/30c049e4-0dd3-429d-b257-f319edb696e0/volumes"
Sep 16 00:52:21.088607 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.088559    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 16 00:52:21.088823 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.088633    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 16 00:52:21.561891 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.561853    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 00:52:21.563175 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.563147    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c049e4-0dd3-429d-b257-f319edb696e0" containerName="maas-api"
Sep 16 00:52:21.563317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.563304    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c049e4-0dd3-429d-b257-f319edb696e0" containerName="maas-api"
Sep 16 00:52:21.563520 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.563506    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c049e4-0dd3-429d-b257-f319edb696e0" containerName="maas-api"
Sep 16 00:52:21.566380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.566362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.569255 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.569234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2fa86d-serving-cert\""
Sep 16 00:52:21.574382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.574354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 00:52:21.738789 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.738755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.738979 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.738795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.738979 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.738913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.738979 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.738938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9mc\" (UniqueName: \"kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.840104 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.840011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.840358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.840336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.840578 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.840558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.840710 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.840694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9mc\" (UniqueName: \"kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.841986 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.841940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.842199 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:21.842183    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-serving-cert: secret "maas-api-e2e-shared-b-w2-2fa86d-serving-cert" not found
Sep 16 00:52:21.842365 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:21.842353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls podName:b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562 nodeName:}" failed. No retries permitted until 2026-09-16 00:52:22.342333914 +0000 UTC m=+1112.503844169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls") pod "maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" (UID: "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562") : secret "maas-api-e2e-shared-b-w2-2fa86d-serving-cert" not found
Sep 16 00:52:21.852335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.852302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9mc\" (UniqueName: \"kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:21.864976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:21.864278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:22.134351 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.134271    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:52:22.136659 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.136636    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 00:52:22.347016 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.346982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:22.349419 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.349395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:22.487666 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.487584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:22.765012 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:22.764983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 00:52:22.766125 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:52:22.766083    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fd31ea_5a1a_4cb3_ba4e_64186d3c5562.slice/crio-fcccc7073030e548931e2cac7dc17694a21ec8539ed6f0e783e0e612749c0d28 WatchSource:0}: Error finding container fcccc7073030e548931e2cac7dc17694a21ec8539ed6f0e783e0e612749c0d28: Status 404 returned error can't find the container with id fcccc7073030e548931e2cac7dc17694a21ec8539ed6f0e783e0e612749c0d28
Sep 16 00:52:23.388842 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:23.388749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" event={"ID":"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562","Type":"ContainerStarted","Data":"00bde1b444424dee2f7a5c53c730082c37974594fb43523c1996909c4d5e815f"}
Sep 16 00:52:23.388842 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:23.388810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" event={"ID":"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562","Type":"ContainerStarted","Data":"fcccc7073030e548931e2cac7dc17694a21ec8539ed6f0e783e0e612749c0d28"}
Sep 16 00:52:23.388842 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:23.388840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:23.407420 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:23.407378    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" podStartSLOduration=2.077322636 podStartE2EDuration="2.407364244s" totalImagesPullingTime="330.041608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:23.407105846 +0000 UTC m=+1113.568616119" watchObservedRunningTime="2026-09-16 00:52:23.407364244 +0000 UTC m=+1113.568874515"
Sep 16 00:52:24.499480 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.499442    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:52:24.502345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.502326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.505057 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.505037    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-69886ef2-serving-cert\""
Sep 16 00:52:24.513753 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.513723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:52:24.670395 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.670359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.670571 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.670435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.670571 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.670495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.670571 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.670518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsczq\" (UniqueName: \"kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.771605 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.771567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.771782 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.771655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.771782 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.771719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.771782 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.771751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsczq\" (UniqueName: \"kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.771941 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:24.771799    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-serving-cert: secret "maas-api-e2e-migrate-69886ef2-serving-cert" not found
Sep 16 00:52:24.771941 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:24.771890    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls podName:548ad33d-a9e1-4392-9f77-1be536e2ac34 nodeName:}" failed. No retries permitted until 2026-09-16 00:52:25.271868837 +0000 UTC m=+1115.433379089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls") pod "maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" (UID: "548ad33d-a9e1-4392-9f77-1be536e2ac34") : secret "maas-api-e2e-migrate-69886ef2-serving-cert" not found
Sep 16 00:52:24.772372 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.772348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.774252 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.774230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:24.782299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:24.782274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsczq\" (UniqueName: \"kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:25.277183 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:25.277147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:25.279658 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:25.279626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") pod \"maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:25.412135 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:25.412100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:25.689556 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:25.689530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:52:25.690896 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:52:25.690861    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548ad33d_a9e1_4392_9f77_1be536e2ac34.slice/crio-90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1 WatchSource:0}: Error finding container 90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1: Status 404 returned error can't find the container with id 90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1
Sep 16 00:52:26.406941 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:26.406910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" event={"ID":"548ad33d-a9e1-4392-9f77-1be536e2ac34","Type":"ContainerStarted","Data":"a081cfc9e885f4e8f2b70d41e67c7fadcc8dab57e75be7387818a5d7000fb17f"}
Sep 16 00:52:26.407162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:26.406994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" event={"ID":"548ad33d-a9e1-4392-9f77-1be536e2ac34","Type":"ContainerStarted","Data":"90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1"}
Sep 16 00:52:26.407162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:26.407025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:52:26.425137 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:26.425090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" podStartSLOduration=2.149646067 podStartE2EDuration="2.425074939s" totalImagesPullingTime="275.428872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:52:26.423792239 +0000 UTC m=+1116.585302510" watchObservedRunningTime="2026-09-16 00:52:26.425074939 +0000 UTC m=+1116.586585210"
Sep 16 00:52:27.386968 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.386930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:52:27.416123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.416042    2577 generic.go:356] "Generic (PLEG): container finished" podID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerID="8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed" exitCode=0
Sep 16 00:52:27.416123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.416091    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"
Sep 16 00:52:27.416336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.416116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerDied","Data":"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"}
Sep 16 00:52:27.416336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.416168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr" event={"ID":"549a874d-ea31-45ea-8358-25f8c0dbd547","Type":"ContainerDied","Data":"ceab478d7783838a479f2135714831b1d4c4977d702deec65755f283b91a7082"}
Sep 16 00:52:27.416336 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.416190    2577 scope.go:117] "RemoveContainer" containerID="8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"
Sep 16 00:52:27.426071 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.426050    2577 scope.go:117] "RemoveContainer" containerID="eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"
Sep 16 00:52:27.434828 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.434806    2577 scope.go:117] "RemoveContainer" containerID="8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"
Sep 16 00:52:27.435167 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:27.435141    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed\": container with ID starting with 8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed not found: ID does not exist" containerID="8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"
Sep 16 00:52:27.435238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.435180    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed"} err="failed to get container status \"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed\": rpc error: code = NotFound desc = could not find container \"8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed\": container with ID starting with 8a18034803cd2cb4a6ffab28ef7a1e651ad3bbd1e1a86ab683f87f37ab1248ed not found: ID does not exist"
Sep 16 00:52:27.435238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.435204    2577 scope.go:117] "RemoveContainer" containerID="eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"
Sep 16 00:52:27.435506 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:52:27.435485    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4\": container with ID starting with eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4 not found: ID does not exist" containerID="eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"
Sep 16 00:52:27.435568 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.435516    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4"} err="failed to get container status \"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4\": rpc error: code = NotFound desc = could not find container \"eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4\": container with ID starting with eb31279ab1e4e1f4a73c2f7d2171c75cd0716017e3d8ac089ca6c9bbc1d6b5b4 not found: ID does not exist"
Sep 16 00:52:27.498964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.498931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499158 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.498983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499158 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499158 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499158 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:27.499371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499371 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzwwr\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr\") pod \"549a874d-ea31-45ea-8358-25f8c0dbd547\" (UID: \"549a874d-ea31-45ea-8358-25f8c0dbd547\") "
Sep 16 00:52:27.499614 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:27.499614 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data" (OuterVolumeSpecName: "istio-data") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:27.499614 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:52:27.499936 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:27.500033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.499985    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.500033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.500010    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.500033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.500025    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.500186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.500040    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.500186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.500053    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a874d-ea31-45ea-8358-25f8c0dbd547-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.501528 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.501497    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 00:52:27.501648 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.501605    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:52:27.501710 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.501695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token" (OuterVolumeSpecName: "istio-token") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:52:27.501763 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.501742    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr" (OuterVolumeSpecName: "kube-api-access-bzwwr") pod "549a874d-ea31-45ea-8358-25f8c0dbd547" (UID: "549a874d-ea31-45ea-8358-25f8c0dbd547"). InnerVolumeSpecName "kube-api-access-bzwwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:52:27.601282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.601241    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.601282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.601280    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwwr\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-kube-api-access-bzwwr\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.601282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.601294    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.601524 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.601307    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a874d-ea31-45ea-8358-25f8c0dbd547-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:52:27.739612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.739580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:52:27.741897 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:27.741870    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-629e17ec-openshift-default-8665578f76-ncnrr"]
Sep 16 00:52:28.334826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:28.334786    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" path="/var/lib/kubelet/pods/549a874d-ea31-45ea-8358-25f8c0dbd547/volumes"
Sep 16 00:52:29.400878 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:29.400846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 00:52:32.421878 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:52:32.421851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:53:03.368761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.368723    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:53:03.369225 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.369029    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" podUID="548ad33d-a9e1-4392-9f77-1be536e2ac34" containerName="maas-api" containerID="cri-o://a081cfc9e885f4e8f2b70d41e67c7fadcc8dab57e75be7387818a5d7000fb17f" gracePeriod=30
Sep 16 00:53:03.625503 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.625420    2577 generic.go:356] "Generic (PLEG): container finished" podID="548ad33d-a9e1-4392-9f77-1be536e2ac34" containerID="a081cfc9e885f4e8f2b70d41e67c7fadcc8dab57e75be7387818a5d7000fb17f" exitCode=0
Sep 16 00:53:03.625503 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.625485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" event={"ID":"548ad33d-a9e1-4392-9f77-1be536e2ac34","Type":"ContainerDied","Data":"a081cfc9e885f4e8f2b70d41e67c7fadcc8dab57e75be7387818a5d7000fb17f"}
Sep 16 00:53:03.625681 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.625549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm" event={"ID":"548ad33d-a9e1-4392-9f77-1be536e2ac34","Type":"ContainerDied","Data":"90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1"}
Sep 16 00:53:03.625681 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.625573    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90bda723f473257bd2b844997abbd1899043f29bdaabe1d65f8aac9170a62bb1"
Sep 16 00:53:03.636915 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.636896    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:53:03.772004 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.771946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca\") pod \"548ad33d-a9e1-4392-9f77-1be536e2ac34\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") "
Sep 16 00:53:03.772195 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.772026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs\") pod \"548ad33d-a9e1-4392-9f77-1be536e2ac34\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") "
Sep 16 00:53:03.772195 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.772121    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") pod \"548ad33d-a9e1-4392-9f77-1be536e2ac34\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") "
Sep 16 00:53:03.772327 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.772211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsczq\" (UniqueName: \"kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq\") pod \"548ad33d-a9e1-4392-9f77-1be536e2ac34\" (UID: \"548ad33d-a9e1-4392-9f77-1be536e2ac34\") "
Sep 16 00:53:03.772327 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.772290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "548ad33d-a9e1-4392-9f77-1be536e2ac34" (UID: "548ad33d-a9e1-4392-9f77-1be536e2ac34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:03.772548 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.772527    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548ad33d-a9e1-4392-9f77-1be536e2ac34-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:03.774368 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.774334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "548ad33d-a9e1-4392-9f77-1be536e2ac34" (UID: "548ad33d-a9e1-4392-9f77-1be536e2ac34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:03.774500 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.774369    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "548ad33d-a9e1-4392-9f77-1be536e2ac34" (UID: "548ad33d-a9e1-4392-9f77-1be536e2ac34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:03.774500 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.774441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq" (OuterVolumeSpecName: "kube-api-access-fsczq") pod "548ad33d-a9e1-4392-9f77-1be536e2ac34" (UID: "548ad33d-a9e1-4392-9f77-1be536e2ac34"). InnerVolumeSpecName "kube-api-access-fsczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:03.873474 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.873432    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:03.873474 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.873470    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548ad33d-a9e1-4392-9f77-1be536e2ac34-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:03.873701 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:03.873484    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsczq\" (UniqueName: \"kubernetes.io/projected/548ad33d-a9e1-4392-9f77-1be536e2ac34-kube-api-access-fsczq\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:04.508558 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.508509    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:04.509199 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509180    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548ad33d-a9e1-4392-9f77-1be536e2ac34" containerName="maas-api"
Sep 16 00:53:04.509199 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509199    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="548ad33d-a9e1-4392-9f77-1be536e2ac34" containerName="maas-api"
Sep 16 00:53:04.509358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509235    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.509358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509244    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.509358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509268    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.509358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509276    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.509849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509373    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.509849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509386    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="548ad33d-a9e1-4392-9f77-1be536e2ac34" containerName="maas-api"
Sep 16 00:53:04.509849 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.509396    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="549a874d-ea31-45ea-8358-25f8c0dbd547" containerName="istio-proxy"
Sep 16 00:53:04.513210 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.513190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.515999 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.515979    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9f35fa48-openshift-default-dockercfg-rbqmp\""
Sep 16 00:53:04.523162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.523132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:04.633614 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.633521    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"
Sep 16 00:53:04.650746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.650713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:53:04.659226 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.659196    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69886ef2-5948bdd4c5-ktqvm"]
Sep 16 00:53:04.681096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681262 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681262 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681262 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kfrt\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681550 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.681550 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.681493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kfrt\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782519 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.782925 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.783024 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.782915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.783091 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.783036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.783149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.783083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.783212 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.783155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.783524 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.783500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.785011 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.784989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.785219 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.785204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.790230 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.790208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.790747 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.790725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kfrt\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt\") pod \"e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") " pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:04.826668 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:04.826636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:05.108208 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.108182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:05.109884 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:53:05.109855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3130bb8_a91c_4a89_83c8_2775fa22f2bb.slice/crio-614914e0ac194372ac3109903f1dce3b89e40470170db93de80ac2bab01bc87b WatchSource:0}: Error finding container 614914e0ac194372ac3109903f1dce3b89e40470170db93de80ac2bab01bc87b: Status 404 returned error can't find the container with id 614914e0ac194372ac3109903f1dce3b89e40470170db93de80ac2bab01bc87b
Sep 16 00:53:05.112190 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.112155    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:05.112290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.112237    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:05.112290 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.112280    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:05.640708 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.640663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerStarted","Data":"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"}
Sep 16 00:53:05.641166 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.640729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerStarted","Data":"614914e0ac194372ac3109903f1dce3b89e40470170db93de80ac2bab01bc87b"}
Sep 16 00:53:05.664161 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.664107    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podStartSLOduration=1.664091378 podStartE2EDuration="1.664091378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:05.662182128 +0000 UTC m=+1155.823692411" watchObservedRunningTime="2026-09-16 00:53:05.664091378 +0000 UTC m=+1155.825601649"
Sep 16 00:53:05.827753 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:05.827719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:06.335189 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:06.335155    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548ad33d-a9e1-4392-9f77-1be536e2ac34" path="/var/lib/kubelet/pods/548ad33d-a9e1-4392-9f77-1be536e2ac34/volumes"
Sep 16 00:53:06.832975 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:06.832928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:07.652879 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:07.652846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:07.653727 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:07.653708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:09.892967 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.892913    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:09.898177 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.898146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:09.900610 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.900584    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9f35fa48-serving-cert\""
Sep 16 00:53:09.907380 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.907355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:09.927396 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.927361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj8l\" (UniqueName: \"kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:09.927535 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.927411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:09.927592 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.927553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:09.927660 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:09.927642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.028311 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.028276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj8l\" (UniqueName: \"kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.028490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.028327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.028490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.028413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.028490 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.028466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.028671 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:10.028585    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-serving-cert: secret "maas-api-e2e-ait-9f35fa48-serving-cert" not found
Sep 16 00:53:10.028671 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:10.028667    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls podName:b377d6c7-9b64-42e8-ba89-3af2e87e87b1 nodeName:}" failed. No retries permitted until 2026-09-16 00:53:10.528645228 +0000 UTC m=+1160.690155480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls") pod "maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" (UID: "b377d6c7-9b64-42e8-ba89-3af2e87e87b1") : secret "maas-api-e2e-ait-9f35fa48-serving-cert" not found
Sep 16 00:53:10.029227 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.029206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.031059 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.031038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.036473 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.036451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj8l\" (UniqueName: \"kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.531965 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.531929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.534608 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.534582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") pod \"maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:10.813260 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:10.813154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:11.088000 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:11.087970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:11.090124 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:53:11.090096    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb377d6c7_9b64_42e8_ba89_3af2e87e87b1.slice/crio-b8e0ef6f3873b0b0b0650d031f05343786a140633d9c148c7fc1671ab8ad3d59 WatchSource:0}: Error finding container b8e0ef6f3873b0b0b0650d031f05343786a140633d9c148c7fc1671ab8ad3d59: Status 404 returned error can't find the container with id b8e0ef6f3873b0b0b0650d031f05343786a140633d9c148c7fc1671ab8ad3d59
Sep 16 00:53:11.683971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:11.683912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" event={"ID":"b377d6c7-9b64-42e8-ba89-3af2e87e87b1","Type":"ContainerStarted","Data":"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"}
Sep 16 00:53:11.684154 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:11.684021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" event={"ID":"b377d6c7-9b64-42e8-ba89-3af2e87e87b1","Type":"ContainerStarted","Data":"b8e0ef6f3873b0b0b0650d031f05343786a140633d9c148c7fc1671ab8ad3d59"}
Sep 16 00:53:11.684154 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:11.684052    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:11.702015 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:11.701967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" podStartSLOduration=2.235368483 podStartE2EDuration="2.701939992s" totalImagesPullingTime="466.571509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:11.699751828 +0000 UTC m=+1161.861262093" watchObservedRunningTime="2026-09-16 00:53:11.701939992 +0000 UTC m=+1161.863450264"
Sep 16 00:53:17.695572 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:17.695541    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:22.327393 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:22.327362    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:53:27.597623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.597584    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:53:27.598274 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.598212    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" podUID="c9bae60a-e806-4940-984d-3b090889a55a" containerName="maas-api" containerID="cri-o://cc8de2803d81ca74e9f519ff31ef7eb2654e29ce4f5b201f10e6c2fe6050d001" gracePeriod=30
Sep 16 00:53:27.782719 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.782689    2577 generic.go:356] "Generic (PLEG): container finished" podID="c9bae60a-e806-4940-984d-3b090889a55a" containerID="cc8de2803d81ca74e9f519ff31ef7eb2654e29ce4f5b201f10e6c2fe6050d001" exitCode=0
Sep 16 00:53:27.782909 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.782765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" event={"ID":"c9bae60a-e806-4940-984d-3b090889a55a","Type":"ContainerDied","Data":"cc8de2803d81ca74e9f519ff31ef7eb2654e29ce4f5b201f10e6c2fe6050d001"}
Sep 16 00:53:27.924445 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.924418    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:53:27.998388 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.998360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm9t4\" (UniqueName: \"kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4\") pod \"c9bae60a-e806-4940-984d-3b090889a55a\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") "
Sep 16 00:53:27.998679 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.998667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca\") pod \"c9bae60a-e806-4940-984d-3b090889a55a\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") "
Sep 16 00:53:27.998996 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.998886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs\") pod \"c9bae60a-e806-4940-984d-3b090889a55a\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") "
Sep 16 00:53:27.999345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.999263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9bae60a-e806-4940-984d-3b090889a55a" (UID: "c9bae60a-e806-4940-984d-3b090889a55a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:27.999345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.999332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") pod \"c9bae60a-e806-4940-984d-3b090889a55a\" (UID: \"c9bae60a-e806-4940-984d-3b090889a55a\") "
Sep 16 00:53:27.999710 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:27.999681    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bae60a-e806-4940-984d-3b090889a55a-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:28.001082 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.001059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4" (OuterVolumeSpecName: "kube-api-access-zm9t4") pod "c9bae60a-e806-4940-984d-3b090889a55a" (UID: "c9bae60a-e806-4940-984d-3b090889a55a"). InnerVolumeSpecName "kube-api-access-zm9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:28.001997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.001925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9bae60a-e806-4940-984d-3b090889a55a" (UID: "c9bae60a-e806-4940-984d-3b090889a55a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:28.002171 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.002147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9bae60a-e806-4940-984d-3b090889a55a" (UID: "c9bae60a-e806-4940-984d-3b090889a55a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:28.101001 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.100933    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:28.101001 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.100996    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9t4\" (UniqueName: \"kubernetes.io/projected/c9bae60a-e806-4940-984d-3b090889a55a-kube-api-access-zm9t4\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:28.101001 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.101011    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bae60a-e806-4940-984d-3b090889a55a-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:28.789881 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.789847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf" event={"ID":"c9bae60a-e806-4940-984d-3b090889a55a","Type":"ContainerDied","Data":"3c0c2e9c593bcf0cdd5abae0030a1ddfb36667c9164eedbb7f0a0e9ad223eefe"}
Sep 16 00:53:28.790366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.789906    2577 scope.go:117] "RemoveContainer" containerID="cc8de2803d81ca74e9f519ff31ef7eb2654e29ce4f5b201f10e6c2fe6050d001"
Sep 16 00:53:28.790366 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.789854    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"
Sep 16 00:53:28.807934 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.807907    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:53:28.811448 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:28.811423    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f7c3b-8554cc84b4-mhmnf"]
Sep 16 00:53:30.335237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:30.335193    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9bae60a-e806-4940-984d-3b090889a55a" path="/var/lib/kubelet/pods/c9bae60a-e806-4940-984d-3b090889a55a/volumes"
Sep 16 00:53:32.819659 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.819587    2577 generic.go:356] "Generic (PLEG): container finished" podID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerID="5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233" exitCode=0
Sep 16 00:53:32.820060 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.819660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerDied","Data":"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"}
Sep 16 00:53:32.820060 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.820050    2577 scope.go:117] "RemoveContainer" containerID="5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"
Sep 16 00:53:32.821176 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.821148    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:32.821241 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.821217    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:32.821286 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:32.821249    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:53:33.826895 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:33.826852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerStarted","Data":"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"}
Sep 16 00:53:34.327799 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.327765    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:53:34.514841 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.514811    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:53:34.515045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.515026    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" containerID="cri-o://486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932" gracePeriod=30
Sep 16 00:53:34.827620 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.827585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:34.828870 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.828840    2577 patch_prober.go:28] interesting pod/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 00:53:34.829034 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:34.828923    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 00:53:35.357406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:35.357366    2577 patch_prober.go:28] interesting pod/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 00:53:35.357675 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:35.357646    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:53:35.827917 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:35.827885    2577 patch_prober.go:28] interesting pod/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 00:53:35.828334 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:35.827939    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 00:53:36.827470 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:36.827433    2577 patch_prober.go:28] interesting pod/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 00:53:36.827641 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:36.827485    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 00:53:37.653218 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:37.653180    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:37.827558 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:37.827522    2577 patch_prober.go:28] interesting pod/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 00:53:37.827733 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:37.827574    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 00:53:38.827369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:38.827335    2577 patch_prober.go:28] interesting pod/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 00:53:38.827761 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:38.827391    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 00:53:39.100920 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.100836    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:39.101127 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.101048    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" podUID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" containerName="maas-api" containerID="cri-o://7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3" gracePeriod=30
Sep 16 00:53:39.407062 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.407035    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:39.520772 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.520733    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs\") pod \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") "
Sep 16 00:53:39.520937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.520813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") pod \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") "
Sep 16 00:53:39.520937 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.520840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca\") pod \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") "
Sep 16 00:53:39.521066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.520986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzj8l\" (UniqueName: \"kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l\") pod \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\" (UID: \"b377d6c7-9b64-42e8-ba89-3af2e87e87b1\") "
Sep 16 00:53:39.521421 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.521286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b377d6c7-9b64-42e8-ba89-3af2e87e87b1" (UID: "b377d6c7-9b64-42e8-ba89-3af2e87e87b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:39.521421 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.521384    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.523291 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.523258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b377d6c7-9b64-42e8-ba89-3af2e87e87b1" (UID: "b377d6c7-9b64-42e8-ba89-3af2e87e87b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:39.523291 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.523269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b377d6c7-9b64-42e8-ba89-3af2e87e87b1" (UID: "b377d6c7-9b64-42e8-ba89-3af2e87e87b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:53:39.523635 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.523619    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l" (OuterVolumeSpecName: "kube-api-access-xzj8l") pod "b377d6c7-9b64-42e8-ba89-3af2e87e87b1" (UID: "b377d6c7-9b64-42e8-ba89-3af2e87e87b1"). InnerVolumeSpecName "kube-api-access-xzj8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:39.622941 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.622905    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.622941 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.622936    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj8l\" (UniqueName: \"kubernetes.io/projected/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-kube-api-access-xzj8l\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.623218 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.622972    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b377d6c7-9b64-42e8-ba89-3af2e87e87b1-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.772890 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.772862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:53:39.824400 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824370    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824448    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824602 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2s7j\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j\") pod \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\" (UID: \"73e41cc9-76ff-4bb7-808b-6a5184ce5f78\") "
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data" (OuterVolumeSpecName: "istio-data") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:39.824926 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.824912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:39.825204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.825080    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.825204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.825100    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.825204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.825113    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.825204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.825123    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.825406 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.825348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:39.827730 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.827702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 00:53:39.828146 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.827881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:39.828358 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.828334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token" (OuterVolumeSpecName: "istio-token") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:39.828462 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.828368    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j" (OuterVolumeSpecName: "kube-api-access-l2s7j") pod "73e41cc9-76ff-4bb7-808b-6a5184ce5f78" (UID: "73e41cc9-76ff-4bb7-808b-6a5184ce5f78"). InnerVolumeSpecName "kube-api-access-l2s7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:39.873105 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.873068    2577 generic.go:356] "Generic (PLEG): container finished" podID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" containerID="7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3" exitCode=0
Sep 16 00:53:39.873283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.873122    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"
Sep 16 00:53:39.873283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.873120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" event={"ID":"b377d6c7-9b64-42e8-ba89-3af2e87e87b1","Type":"ContainerDied","Data":"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"}
Sep 16 00:53:39.873283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.873176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq" event={"ID":"b377d6c7-9b64-42e8-ba89-3af2e87e87b1","Type":"ContainerDied","Data":"b8e0ef6f3873b0b0b0650d031f05343786a140633d9c148c7fc1671ab8ad3d59"}
Sep 16 00:53:39.873283 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.873198    2577 scope.go:117] "RemoveContainer" containerID="7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"
Sep 16 00:53:39.875734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.875594    2577 generic.go:356] "Generic (PLEG): container finished" podID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerID="486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932" exitCode=0
Sep 16 00:53:39.875734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.875640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerDied","Data":"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"}
Sep 16 00:53:39.875734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.875658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"
Sep 16 00:53:39.875734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.875682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8" event={"ID":"73e41cc9-76ff-4bb7-808b-6a5184ce5f78","Type":"ContainerDied","Data":"51dee0079f398fc4e6abea7de67f1375b38a1f5f25e6e663a09e91d3bcf5e132"}
Sep 16 00:53:39.883938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.883921    2577 scope.go:117] "RemoveContainer" containerID="7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"
Sep 16 00:53:39.884253 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:39.884235    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3\": container with ID starting with 7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3 not found: ID does not exist" containerID="7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"
Sep 16 00:53:39.884317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.884260    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3"} err="failed to get container status \"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3\": rpc error: code = NotFound desc = could not find container \"7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3\": container with ID starting with 7cba0987c04007fc36ac706d7a44be14f594f43aed58119e711e23c799a1bbc3 not found: ID does not exist"
Sep 16 00:53:39.884317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.884279    2577 scope.go:117] "RemoveContainer" containerID="486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"
Sep 16 00:53:39.895891 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.895861    2577 scope.go:117] "RemoveContainer" containerID="0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"
Sep 16 00:53:39.904359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.904303    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:39.909868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.909835    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9f35fa48-745d4fb9b-crshq"]
Sep 16 00:53:39.912346 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.912323    2577 scope.go:117] "RemoveContainer" containerID="486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"
Sep 16 00:53:39.912746 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:39.912718    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932\": container with ID starting with 486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932 not found: ID does not exist" containerID="486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"
Sep 16 00:53:39.912860 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.912757    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932"} err="failed to get container status \"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932\": rpc error: code = NotFound desc = could not find container \"486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932\": container with ID starting with 486a85d5bc7c887d363125892309870be724afdcd8eaee0297b68113bd59f932 not found: ID does not exist"
Sep 16 00:53:39.912860 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.912784    2577 scope.go:117] "RemoveContainer" containerID="0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"
Sep 16 00:53:39.913236 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:39.913208    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d\": container with ID starting with 0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d not found: ID does not exist" containerID="0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"
Sep 16 00:53:39.913341 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.913247    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d"} err="failed to get container status \"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d\": rpc error: code = NotFound desc = could not find container \"0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d\": container with ID starting with 0fca5a0799700285c10a6b80dd3e1b7798033f51c254d668cccddf7cda1a839d not found: ID does not exist"
Sep 16 00:53:39.917175 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.917152    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:53:39.922516 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.922495    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f7c3b-openshift-default-679dbb4fb8-gvqx8"]
Sep 16 00:53:39.925993 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.925973    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.926078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.926009    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.926078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.926024    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.926078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.926035    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2s7j\" (UniqueName: \"kubernetes.io/projected/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-kube-api-access-l2s7j\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.926078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.926048    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e41cc9-76ff-4bb7-808b-6a5184ce5f78-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:39.947878 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.947778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:39.950162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:39.950142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:40.334910 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:40.334858    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" path="/var/lib/kubelet/pods/73e41cc9-76ff-4bb7-808b-6a5184ce5f78/volumes"
Sep 16 00:53:40.335508 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:40.335486    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" path="/var/lib/kubelet/pods/b377d6c7-9b64-42e8-ba89-3af2e87e87b1/volumes"
Sep 16 00:53:40.575646 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:40.575610    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:40.883749 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:40.883718    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" secret="" err="secret \"e2e-ait-9f35fa48-openshift-default-dockercfg-rbqmp\" not found"
Sep 16 00:53:41.889701 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:41.889659    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy" containerID="cri-o://a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846" gracePeriod=30
Sep 16 00:53:45.312027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.311991    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:53:45.312816 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.312816 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312816    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312839    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312848    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9bae60a-e806-4940-984d-3b090889a55a" containerName="maas-api"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312875    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9bae60a-e806-4940-984d-3b090889a55a" containerName="maas-api"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312894    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" containerName="maas-api"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.312902    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" containerName="maas-api"
Sep 16 00:53:45.313029 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.313026    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b377d6c7-9b64-42e8-ba89-3af2e87e87b1" containerName="maas-api"
Sep 16 00:53:45.313386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.313042    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9bae60a-e806-4940-984d-3b090889a55a" containerName="maas-api"
Sep 16 00:53:45.313386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.313054    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.313386 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.313061    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e41cc9-76ff-4bb7-808b-6a5184ce5f78" containerName="istio-proxy"
Sep 16 00:53:45.321306 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.321282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.325010 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.323915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-68e143c2-serving-cert\""
Sep 16 00:53:45.326475 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.326447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:53:45.484783 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.484752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.485015 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.484834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spfdn\" (UniqueName: \"kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.485015 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.484859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.485015 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.484890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.585610 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.585510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.585610 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.585593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spfdn\" (UniqueName: \"kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.585831 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.585628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.585831 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.585666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.585831 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:45.585801    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-serving-cert: secret "maas-api-e2e-derive-68e143c2-serving-cert" not found
Sep 16 00:53:45.585975 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:45.585870    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls podName:4a24fec6-4dda-4cce-9f75-04caf6a4b7c5 nodeName:}" failed. No retries permitted until 2026-09-16 00:53:46.085850167 +0000 UTC m=+1196.247360418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls") pod "maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" (UID: "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5") : secret "maas-api-e2e-derive-68e143c2-serving-cert" not found
Sep 16 00:53:45.586315 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.586296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.588255 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.588238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:45.594185 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:45.594163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spfdn\" (UniqueName: \"kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:46.090477 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:46.090442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:46.093090 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:46.093060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") pod \"maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:46.236400 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:46.236359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:46.512586 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:46.512561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:53:46.513983 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:53:46.513943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a24fec6_4dda_4cce_9f75_04caf6a4b7c5.slice/crio-632ee752d4eb2f6fa398cfeb362fd04f2901ba2c918ffef11b4c8d3b11078bcd WatchSource:0}: Error finding container 632ee752d4eb2f6fa398cfeb362fd04f2901ba2c918ffef11b4c8d3b11078bcd: Status 404 returned error can't find the container with id 632ee752d4eb2f6fa398cfeb362fd04f2901ba2c918ffef11b4c8d3b11078bcd
Sep 16 00:53:46.919149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:46.919120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" event={"ID":"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5","Type":"ContainerStarted","Data":"632ee752d4eb2f6fa398cfeb362fd04f2901ba2c918ffef11b4c8d3b11078bcd"}
Sep 16 00:53:47.154121 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.154088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:47.304047 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304189    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304231 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kfrt\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:47.304573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:47.304573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:47.304573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:47.304789 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert\") pod \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\" (UID: \"e3130bb8-a91c-4a89-83c8-2775fa22f2bb\") "
Sep 16 00:53:47.304991 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.304936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:53:47.305123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.305045    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.305123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.305063    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.305123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.305074    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.305123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.305084    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.305123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.305096    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.306485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.306466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:53:47.306575 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.306490    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:47.307006 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.306992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 00:53:47.307073 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.307029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt" (OuterVolumeSpecName: "kube-api-access-7kfrt") pod "e3130bb8-a91c-4a89-83c8-2775fa22f2bb" (UID: "e3130bb8-a91c-4a89-83c8-2775fa22f2bb"). InnerVolumeSpecName "kube-api-access-7kfrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:53:47.406317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.406241    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.406317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.406266    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.406317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.406275    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.406317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.406284    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kfrt\" (UniqueName: \"kubernetes.io/projected/e3130bb8-a91c-4a89-83c8-2775fa22f2bb-kube-api-access-7kfrt\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:53:47.926005 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.925973    2577 generic.go:356] "Generic (PLEG): container finished" podID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerID="a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846" exitCode=0
Sep 16 00:53:47.926407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.926025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"
Sep 16 00:53:47.926407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.926052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerDied","Data":"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"}
Sep 16 00:53:47.926407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.926125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8" event={"ID":"e3130bb8-a91c-4a89-83c8-2775fa22f2bb","Type":"ContainerDied","Data":"614914e0ac194372ac3109903f1dce3b89e40470170db93de80ac2bab01bc87b"}
Sep 16 00:53:47.926407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.926161    2577 scope.go:117] "RemoveContainer" containerID="a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"
Sep 16 00:53:47.927829 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.927808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" event={"ID":"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5","Type":"ContainerStarted","Data":"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"}
Sep 16 00:53:47.927968 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.927930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:53:47.936550 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.936534    2577 scope.go:117] "RemoveContainer" containerID="5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"
Sep 16 00:53:47.945423 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.945401    2577 scope.go:117] "RemoveContainer" containerID="a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"
Sep 16 00:53:47.945801 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:47.945766    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846\": container with ID starting with a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846 not found: ID does not exist" containerID="a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"
Sep 16 00:53:47.945913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.945797    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846"} err="failed to get container status \"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846\": rpc error: code = NotFound desc = could not find container \"a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846\": container with ID starting with a020456a8fec6fbefc56133b1f80aeeaff32737c975d117b107f64c9fe35b846 not found: ID does not exist"
Sep 16 00:53:47.945913 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.945821    2577 scope.go:117] "RemoveContainer" containerID="5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"
Sep 16 00:53:47.946219 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:53:47.946187    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233\": container with ID starting with 5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233 not found: ID does not exist" containerID="5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"
Sep 16 00:53:47.946335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.946215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233"} err="failed to get container status \"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233\": rpc error: code = NotFound desc = could not find container \"5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233\": container with ID starting with 5513dbfe8eee86b5080200c9f6a51ee6346553b76351e26ca3ce408b7d73e233 not found: ID does not exist"
Sep 16 00:53:47.947533 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.947495    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" podStartSLOduration=2.609215087 podStartE2EDuration="2.947483574s" totalImagesPullingTime="338.268487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:53:47.94535869 +0000 UTC m=+1198.106868967" watchObservedRunningTime="2026-09-16 00:53:47.947483574 +0000 UTC m=+1198.108993846"
Sep 16 00:53:47.960149 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.960127    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:47.963120 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:47.963100    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9f35fa48-openshift-default-9cc786984-nvhs8"]
Sep 16 00:53:48.334226 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:48.334195    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" path="/var/lib/kubelet/pods/e3130bb8-a91c-4a89-83c8-2775fa22f2bb/volumes"
Sep 16 00:53:50.387195 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:50.387167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:53:50.391987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:50.391942    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:53:53.940746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:53:53.940716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:54:15.602664 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.602621    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:54:15.603164 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.602831    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" podUID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" containerName="maas-api" containerID="cri-o://58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5" gracePeriod=30
Sep 16 00:54:15.888454 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.888412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:54:15.986982 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.986920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") pod \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") "
Sep 16 00:54:15.987187 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.986999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spfdn\" (UniqueName: \"kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn\") pod \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") "
Sep 16 00:54:15.987187 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.987107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca\") pod \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") "
Sep 16 00:54:15.987187 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.987126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs\") pod \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\" (UID: \"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5\") "
Sep 16 00:54:15.987536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.987508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" (UID: "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:54:15.989360 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.989323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" (UID: "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:54:15.989474 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.989396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn" (OuterVolumeSpecName: "kube-api-access-spfdn") pod "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" (UID: "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5"). InnerVolumeSpecName "kube-api-access-spfdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:54:15.989621 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:15.989607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" (UID: "4a24fec6-4dda-4cce-9f75-04caf6a4b7c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:54:16.086786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.086750    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" containerID="58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5" exitCode=0
Sep 16 00:54:16.087045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.086793    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"
Sep 16 00:54:16.087045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.086833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" event={"ID":"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5","Type":"ContainerDied","Data":"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"}
Sep 16 00:54:16.087045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.086903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw" event={"ID":"4a24fec6-4dda-4cce-9f75-04caf6a4b7c5","Type":"ContainerDied","Data":"632ee752d4eb2f6fa398cfeb362fd04f2901ba2c918ffef11b4c8d3b11078bcd"}
Sep 16 00:54:16.087045 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.086940    2577 scope.go:117] "RemoveContainer" containerID="58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"
Sep 16 00:54:16.087862 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.087844    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:54:16.087932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.087865    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:54:16.087932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.087875    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:54:16.087932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.087883    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spfdn\" (UniqueName: \"kubernetes.io/projected/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5-kube-api-access-spfdn\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:54:16.099462 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.099066    2577 scope.go:117] "RemoveContainer" containerID="58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"
Sep 16 00:54:16.099622 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:54:16.099598    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5\": container with ID starting with 58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5 not found: ID does not exist" containerID="58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"
Sep 16 00:54:16.099701 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.099639    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5"} err="failed to get container status \"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5\": rpc error: code = NotFound desc = could not find container \"58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5\": container with ID starting with 58dbabbfb0ad5987fec2ba0583f004cdde0155a2d0d0153145ca7e8e134818b5 not found: ID does not exist"
Sep 16 00:54:16.115430 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.115336    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:54:16.129995 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.129935    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68e143c2-6bfd58695c-grwdw"]
Sep 16 00:54:16.344172 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:16.344131    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" path="/var/lib/kubelet/pods/4a24fec6-4dda-4cce-9f75-04caf6a4b7c5/volumes"
Sep 16 00:54:37.327482 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:37.327438    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:54:50.330455 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:54:50.330359    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:55:12.340664 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:12.340623    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:55:23.159984 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.159927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:55:23.160517 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.160256    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" podUID="ade09380-f47e-4ac4-a994-74e4992fb3ea" containerName="maas-api" containerID="cri-o://a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc" gracePeriod=30
Sep 16 00:55:23.427911 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.427839    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:55:23.476081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.476041    2577 generic.go:356] "Generic (PLEG): container finished" podID="ade09380-f47e-4ac4-a994-74e4992fb3ea" containerID="a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc" exitCode=0
Sep 16 00:55:23.476293 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.476090    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"
Sep 16 00:55:23.476293 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.476130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" event={"ID":"ade09380-f47e-4ac4-a994-74e4992fb3ea","Type":"ContainerDied","Data":"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"}
Sep 16 00:55:23.476293 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.476202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58" event={"ID":"ade09380-f47e-4ac4-a994-74e4992fb3ea","Type":"ContainerDied","Data":"7c3e7d45a7684d92d7861cd66e9c3842b1eab3fb9c392b6166c721fc272a67a1"}
Sep 16 00:55:23.476293 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.476236    2577 scope.go:117] "RemoveContainer" containerID="a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"
Sep 16 00:55:23.488683 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.488659    2577 scope.go:117] "RemoveContainer" containerID="a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"
Sep 16 00:55:23.489175 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:55:23.489149    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc\": container with ID starting with a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc not found: ID does not exist" containerID="a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"
Sep 16 00:55:23.489298 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.489182    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc"} err="failed to get container status \"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc\": rpc error: code = NotFound desc = could not find container \"a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc\": container with ID starting with a378e66ea22a371a50dc6494d0536fe71409cd3f9fc2ed792f38ffd280a59bcc not found: ID does not exist"
Sep 16 00:55:23.533786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.533752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs\") pod \"ade09380-f47e-4ac4-a994-74e4992fb3ea\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") "
Sep 16 00:55:23.534011 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.533882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfttw\" (UniqueName: \"kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw\") pod \"ade09380-f47e-4ac4-a994-74e4992fb3ea\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") "
Sep 16 00:55:23.534011 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.533932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca\") pod \"ade09380-f47e-4ac4-a994-74e4992fb3ea\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") "
Sep 16 00:55:23.534153 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.534040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") pod \"ade09380-f47e-4ac4-a994-74e4992fb3ea\" (UID: \"ade09380-f47e-4ac4-a994-74e4992fb3ea\") "
Sep 16 00:55:23.534433 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.534396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ade09380-f47e-4ac4-a994-74e4992fb3ea" (UID: "ade09380-f47e-4ac4-a994-74e4992fb3ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:55:23.536345 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.536309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw" (OuterVolumeSpecName: "kube-api-access-lfttw") pod "ade09380-f47e-4ac4-a994-74e4992fb3ea" (UID: "ade09380-f47e-4ac4-a994-74e4992fb3ea"). InnerVolumeSpecName "kube-api-access-lfttw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:23.536826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.536798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ade09380-f47e-4ac4-a994-74e4992fb3ea" (UID: "ade09380-f47e-4ac4-a994-74e4992fb3ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:55:23.536886 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.536821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ade09380-f47e-4ac4-a994-74e4992fb3ea" (UID: "ade09380-f47e-4ac4-a994-74e4992fb3ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:55:23.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.635191    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:23.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.635228    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfttw\" (UniqueName: \"kubernetes.io/projected/ade09380-f47e-4ac4-a994-74e4992fb3ea-kube-api-access-lfttw\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:23.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.635238    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade09380-f47e-4ac4-a994-74e4992fb3ea-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:23.635238 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.635249    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade09380-f47e-4ac4-a994-74e4992fb3ea-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:23.802821 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.802764    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:55:23.806661 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:23.806629    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5270e2-5f4bd8cc55-7dt58"]
Sep 16 00:55:24.335310 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:24.335277    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade09380-f47e-4ac4-a994-74e4992fb3ea" path="/var/lib/kubelet/pods/ade09380-f47e-4ac4-a994-74e4992fb3ea/volumes"
Sep 16 00:55:27.361898 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.361846    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" podUID="1520e620-b82e-4b1c-8e62-692f64a790ec" containerName="main" containerID="cri-o://18652dd5d4e08357d0e6b98e421c3effe89259e21013a6366317f1c917b23fe9" gracePeriod=45
Sep 16 00:55:27.501478 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.501444    2577 generic.go:356] "Generic (PLEG): container finished" podID="1520e620-b82e-4b1c-8e62-692f64a790ec" containerID="18652dd5d4e08357d0e6b98e421c3effe89259e21013a6366317f1c917b23fe9" exitCode=0
Sep 16 00:55:27.501661 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.501503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" event={"ID":"1520e620-b82e-4b1c-8e62-692f64a790ec","Type":"ContainerDied","Data":"18652dd5d4e08357d0e6b98e421c3effe89259e21013a6366317f1c917b23fe9"}
Sep 16 00:55:27.638285 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.638253    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:55:27.777740 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.777964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.777964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sddb\" (UniqueName: \"kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.777964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.777964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.777964 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.777964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir\") pod \"1520e620-b82e-4b1c-8e62-692f64a790ec\" (UID: \"1520e620-b82e-4b1c-8e62-692f64a790ec\") "
Sep 16 00:55:27.778523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.778489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:55:27.778668 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.778513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home" (OuterVolumeSpecName: "home") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:55:27.778743 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.778664    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:55:27.780741 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.780697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb" (OuterVolumeSpecName: "kube-api-access-9sddb") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "kube-api-access-9sddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:55:27.780862 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.780789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm" (OuterVolumeSpecName: "dshm") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:55:27.780980 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.780934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1520e620-b82e-4b1c-8e62-692f64a790ec" (UID: "1520e620-b82e-4b1c-8e62-692f64a790ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879011    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-dshm\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879048    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1520e620-b82e-4b1c-8e62-692f64a790ec-tls-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sddb\" (UniqueName: \"kubernetes.io/projected/1520e620-b82e-4b1c-8e62-692f64a790ec-kube-api-access-9sddb\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879068    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-home\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879077    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-model-cache\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:27.879113 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:27.879084    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1520e620-b82e-4b1c-8e62-692f64a790ec-tmp-dir\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:55:28.508891 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:28.508847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r" event={"ID":"1520e620-b82e-4b1c-8e62-692f64a790ec","Type":"ContainerDied","Data":"64d1bac88cfcc7bbce8d8e3453c3647e3a4e981b6fdb73f82074c3051d61c790"}
Sep 16 00:55:28.508891 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:28.508895    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"
Sep 16 00:55:28.509404 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:28.508918    2577 scope.go:117] "RemoveContainer" containerID="18652dd5d4e08357d0e6b98e421c3effe89259e21013a6366317f1c917b23fe9"
Sep 16 00:55:28.525319 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:28.525284    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:55:28.528569 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:28.528542    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5270e2/distinct-2-w0-5270e2-kserve-7f7d7cf5dc-kcf2r"]
Sep 16 00:55:30.335516 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:30.335478    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1520e620-b82e-4b1c-8e62-692f64a790ec" path="/var/lib/kubelet/pods/1520e620-b82e-4b1c-8e62-692f64a790ec/volumes"
Sep 16 00:55:47.327775 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:47.327735    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:55:51.554809 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.554765    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:55:51.555468 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555448    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade09380-f47e-4ac4-a994-74e4992fb3ea" containerName="maas-api"
Sep 16 00:55:51.555532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555470    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade09380-f47e-4ac4-a994-74e4992fb3ea" containerName="maas-api"
Sep 16 00:55:51.555532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555484    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555492    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555499    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" containerName="maas-api"
Sep 16 00:55:51.555532 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555530    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" containerName="maas-api"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555547    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555553    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555560    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1520e620-b82e-4b1c-8e62-692f64a790ec" containerName="main"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555565    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1520e620-b82e-4b1c-8e62-692f64a790ec" containerName="main"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555644    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1520e620-b82e-4b1c-8e62-692f64a790ec" containerName="main"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3130bb8-a91c-4a89-83c8-2775fa22f2bb" containerName="istio-proxy"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555660    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade09380-f47e-4ac4-a994-74e4992fb3ea" containerName="maas-api"
Sep 16 00:55:51.555709 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.555666    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a24fec6-4dda-4cce-9f75-04caf6a4b7c5" containerName="maas-api"
Sep 16 00:55:51.560095 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.560069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.562699 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.562666    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-228a5f03-openshift-default-dockercfg-d8kff\""
Sep 16 00:55:51.569096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.569057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:55:51.606192 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4gw\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606390 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.606641 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.606404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707316 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4gw\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707541 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.707932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.707919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.708204 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.708176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.708359 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.708332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.708427 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.708384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.710281 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.710256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.710392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.710307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.715521 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.715495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4gw\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.715983 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.715942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token\") pod \"e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") " pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:51.873890 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:51.873776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:52.155683 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.155651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:55:52.158113 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:55:52.158071    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e84981f_1ed5_409e_ab37_670c43190ca1.slice/crio-1217ae7edda8afbbd41d2ed90beb32e31dfdb7d1b24832cd8505adc1ca04a40d WatchSource:0}: Error finding container 1217ae7edda8afbbd41d2ed90beb32e31dfdb7d1b24832cd8505adc1ca04a40d: Status 404 returned error can't find the container with id 1217ae7edda8afbbd41d2ed90beb32e31dfdb7d1b24832cd8505adc1ca04a40d
Sep 16 00:55:52.161082 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.161041    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:52.161162 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.161135    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:52.161197 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.161186    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:55:52.654907 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.654859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerStarted","Data":"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"}
Sep 16 00:55:52.655442 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.655004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerStarted","Data":"1217ae7edda8afbbd41d2ed90beb32e31dfdb7d1b24832cd8505adc1ca04a40d"}
Sep 16 00:55:52.677317 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.677232    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podStartSLOduration=1.6772079469999999 podStartE2EDuration="1.677207947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:52.673306356 +0000 UTC m=+1322.834816629" watchObservedRunningTime="2026-09-16 00:55:52.677207947 +0000 UTC m=+1322.838718220"
Sep 16 00:55:52.875008 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:52.874934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:53.880060 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:53.880024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:54.668252 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:54.668215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:54.669147 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:54.669124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:55:56.597096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.597027    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:55:56.601848 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.601814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.604335 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.604294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-228a5f03-serving-cert\""
Sep 16 00:55:56.611818 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.610859    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:55:56.656536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.656476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.656754 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.656618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvg6z\" (UniqueName: \"kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.656754 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.656669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.656754 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.656704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.758114 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.758072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvg6z\" (UniqueName: \"kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.758304 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.758228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.758369 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.758300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.758542 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.758521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.758695 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:55:56.758677    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-serving-cert: secret "maas-api-e2e-mt-228a5f03-serving-cert" not found
Sep 16 00:55:56.758783 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:55:56.758732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls podName:f6834005-c5fc-4643-ac4a-302ac0f6ffed nodeName:}" failed. No retries permitted until 2026-09-16 00:55:57.258717293 +0000 UTC m=+1327.420227543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls") pod "maas-api-e2e-mt-228a5f03-558d945984-dsp2j" (UID: "f6834005-c5fc-4643-ac4a-302ac0f6ffed") : secret "maas-api-e2e-mt-228a5f03-serving-cert" not found
Sep 16 00:55:56.759042 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.759023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.761227 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.761205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:56.766936 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:56.766904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvg6z\" (UniqueName: \"kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:57.264573 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:57.264526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:57.267310 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:57.267284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") pod \"maas-api-e2e-mt-228a5f03-558d945984-dsp2j\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:57.518507 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:57.518415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:57.795753 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:57.795706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:55:57.798745 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:55:57.798709    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6834005_c5fc_4643_ac4a_302ac0f6ffed.slice/crio-78bb8f394d8bf6e718023e6ef50ed8421ecb1f9634ae4e81874145e0c2adfc2b WatchSource:0}: Error finding container 78bb8f394d8bf6e718023e6ef50ed8421ecb1f9634ae4e81874145e0c2adfc2b: Status 404 returned error can't find the container with id 78bb8f394d8bf6e718023e6ef50ed8421ecb1f9634ae4e81874145e0c2adfc2b
Sep 16 00:55:57.800817 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:57.800794    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 00:55:58.691685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:58.691651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" event={"ID":"f6834005-c5fc-4643-ac4a-302ac0f6ffed","Type":"ContainerStarted","Data":"65c83e179cc00a11b88d42ac51c95cb21d6e0b0a8a0b7ccb23f088b25c15240a"}
Sep 16 00:55:58.691880 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:58.691708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" event={"ID":"f6834005-c5fc-4643-ac4a-302ac0f6ffed","Type":"ContainerStarted","Data":"78bb8f394d8bf6e718023e6ef50ed8421ecb1f9634ae4e81874145e0c2adfc2b"}
Sep 16 00:55:58.691880 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:58.691854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:55:58.709724 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:55:58.709665    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" podStartSLOduration=2.370511612 podStartE2EDuration="2.709646339s" totalImagesPullingTime="339.134727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:55:58.707565883 +0000 UTC m=+1328.869076155" watchObservedRunningTime="2026-09-16 00:55:58.709646339 +0000 UTC m=+1328.871156611"
Sep 16 00:56:04.704069 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:04.704034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:56:10.711184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:10.711137    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 00:56:10.711659 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:10.711203    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 00:56:11.328173 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:11.328130    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:56:12.784027 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.783985    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerID="31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4" exitCode=0
Sep 16 00:56:12.784535 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.784062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerDied","Data":"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"}
Sep 16 00:56:12.784535 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.784495    2577 scope.go:117] "RemoveContainer" containerID="31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"
Sep 16 00:56:12.785606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.785581    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:12.785670 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.785651    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:12.785708 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:12.785684    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:13.792399 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:13.792356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerStarted","Data":"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"}
Sep 16 00:56:13.874815 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:13.874773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:13.875125 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:13.875101    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 00:56:13.875186 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:13.875167    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 00:56:14.874742 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:14.874705    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 00:56:14.875202 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:14.874765    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 00:56:15.874928 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:15.874882    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 00:56:15.875400 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:15.874963    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 00:56:16.874802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:16.874753    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 00:56:16.875051 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:16.874823    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 00:56:17.875126 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:17.875078    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 00:56:17.875675 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:17.875150    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 00:56:19.084050 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:19.084004    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:19.084612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:19.084429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:19.087546 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:19.087523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:37.781612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:37.781571    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:56:37.782134 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:37.781859    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" podUID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" containerName="maas-api" containerID="cri-o://65c83e179cc00a11b88d42ac51c95cb21d6e0b0a8a0b7ccb23f088b25c15240a" gracePeriod=30
Sep 16 00:56:37.958558 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:37.958521    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" containerID="65c83e179cc00a11b88d42ac51c95cb21d6e0b0a8a0b7ccb23f088b25c15240a" exitCode=0
Sep 16 00:56:37.958750 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:37.958584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" event={"ID":"f6834005-c5fc-4643-ac4a-302ac0f6ffed","Type":"ContainerDied","Data":"65c83e179cc00a11b88d42ac51c95cb21d6e0b0a8a0b7ccb23f088b25c15240a"}
Sep 16 00:56:38.054055 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.054024    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:56:38.188546 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.188509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvg6z\" (UniqueName: \"kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z\") pod \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") "
Sep 16 00:56:38.188546 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.188553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") pod \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") "
Sep 16 00:56:38.188802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.188609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs\") pod \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") "
Sep 16 00:56:38.188802 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.188631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca\") pod \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\" (UID: \"f6834005-c5fc-4643-ac4a-302ac0f6ffed\") "
Sep 16 00:56:38.189110 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.189081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6834005-c5fc-4643-ac4a-302ac0f6ffed" (UID: "f6834005-c5fc-4643-ac4a-302ac0f6ffed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:56:38.190940 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.190908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6834005-c5fc-4643-ac4a-302ac0f6ffed" (UID: "f6834005-c5fc-4643-ac4a-302ac0f6ffed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:56:38.190940 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.190923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6834005-c5fc-4643-ac4a-302ac0f6ffed" (UID: "f6834005-c5fc-4643-ac4a-302ac0f6ffed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 00:56:38.191100 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.190923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z" (OuterVolumeSpecName: "kube-api-access-pvg6z") pod "f6834005-c5fc-4643-ac4a-302ac0f6ffed" (UID: "f6834005-c5fc-4643-ac4a-302ac0f6ffed"). InnerVolumeSpecName "kube-api-access-pvg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:56:38.289751 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.289643    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvg6z\" (UniqueName: \"kubernetes.io/projected/f6834005-c5fc-4643-ac4a-302ac0f6ffed-kube-api-access-pvg6z\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:38.289751 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.289691    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:38.289751 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.289701    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6834005-c5fc-4643-ac4a-302ac0f6ffed-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:38.289751 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.289710    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6834005-c5fc-4643-ac4a-302ac0f6ffed-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:38.966827 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.966798    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"
Sep 16 00:56:38.967319 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.966828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j" event={"ID":"f6834005-c5fc-4643-ac4a-302ac0f6ffed","Type":"ContainerDied","Data":"78bb8f394d8bf6e718023e6ef50ed8421ecb1f9634ae4e81874145e0c2adfc2b"}
Sep 16 00:56:38.967319 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.966913    2577 scope.go:117] "RemoveContainer" containerID="65c83e179cc00a11b88d42ac51c95cb21d6e0b0a8a0b7ccb23f088b25c15240a"
Sep 16 00:56:38.985372 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.985337    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:56:38.988876 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:38.988845    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-228a5f03-558d945984-dsp2j"]
Sep 16 00:56:40.334933 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:40.334892    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" path="/var/lib/kubelet/pods/f6834005-c5fc-4643-ac4a-302ac0f6ffed/volumes"
Sep 16 00:56:44.311392 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:44.311343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:56:44.311868 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:44.311627    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" containerID="cri-o://8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d" gracePeriod=30
Sep 16 00:56:49.086883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.086841    2577 patch_prober.go:28] interesting pod/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 00:56:49.087314 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.086922    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 00:56:49.594670 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.594586    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:49.699428 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699606 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh4gw\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert\") pod \"0e84981f-1ed5-409e-ab37-670c43190ca1\" (UID: \"0e84981f-1ed5-409e-ab37-670c43190ca1\") "
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:56:49.699908 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.699868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:56:49.700337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700050    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:56:49.700337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700164    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.700337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700183    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.700337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700199    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.700337 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700212    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.700553 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.700472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 00:56:49.701889 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.701860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:56:49.702019 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.701886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw" (OuterVolumeSpecName: "kube-api-access-qh4gw") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "kube-api-access-qh4gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:56:49.702346 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.702320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 00:56:49.702346 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.702318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e84981f-1ed5-409e-ab37-670c43190ca1" (UID: "0e84981f-1ed5-409e-ab37-670c43190ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 00:56:49.801209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.801169    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh4gw\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-kube-api-access-qh4gw\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.801209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.801203    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.801209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.801213    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e84981f-1ed5-409e-ab37-670c43190ca1-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.801209 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.801221    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.801484 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.801229    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e84981f-1ed5-409e-ab37-670c43190ca1-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:49.857287 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9361-819dfa"]
Sep 16 00:56:49.857714 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:49.857773 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857715    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:49.857773 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857737    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" containerName="maas-api"
Sep 16 00:56:49.857773 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857743    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" containerName="maas-api"
Sep 16 00:56:49.857867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:49.857867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857815    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:49.857867 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.857823    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6834005-c5fc-4643-ac4a-302ac0f6ffed" containerName="maas-api"
Sep 16 00:56:49.861140 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.861124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:49.863536 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.863515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qndhv\""
Sep 16 00:56:49.866938 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:49.866914    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9361-819dfa"]
Sep 16 00:56:50.003444 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.003349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ddx\" (UniqueName: \"kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx\") pod \"test-curl-9361-819dfa\" (UID: \"7baa3bf7-ac9c-4801-91e6-05875ad4df64\") " pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:50.038581 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.038551    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerID="8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d" exitCode=0
Sep 16 00:56:50.038760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.038599    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"
Sep 16 00:56:50.038760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.038629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerDied","Data":"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"}
Sep 16 00:56:50.038760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.038684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc" event={"ID":"0e84981f-1ed5-409e-ab37-670c43190ca1","Type":"ContainerDied","Data":"1217ae7edda8afbbd41d2ed90beb32e31dfdb7d1b24832cd8505adc1ca04a40d"}
Sep 16 00:56:50.038760 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.038712    2577 scope.go:117] "RemoveContainer" containerID="8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"
Sep 16 00:56:50.048930 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.048910    2577 scope.go:117] "RemoveContainer" containerID="31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"
Sep 16 00:56:50.057488 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.057471    2577 scope.go:117] "RemoveContainer" containerID="8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"
Sep 16 00:56:50.057873 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:56:50.057845    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d\": container with ID starting with 8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d not found: ID does not exist" containerID="8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"
Sep 16 00:56:50.058018 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.057882    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d"} err="failed to get container status \"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d\": rpc error: code = NotFound desc = could not find container \"8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d\": container with ID starting with 8315b9d2878ca4376f15bb3c1b139217a316033a9ff34f7ce18213e07fe4936d not found: ID does not exist"
Sep 16 00:56:50.058018 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.057914    2577 scope.go:117] "RemoveContainer" containerID="31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"
Sep 16 00:56:50.058330 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:56:50.058299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4\": container with ID starting with 31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4 not found: ID does not exist" containerID="31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"
Sep 16 00:56:50.058441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.058337    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4"} err="failed to get container status \"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4\": rpc error: code = NotFound desc = could not find container \"31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4\": container with ID starting with 31f84609cb9885d2e526dd1ad38c47160d7181629172d8ff93a20c45f2f2dbb4 not found: ID does not exist"
Sep 16 00:56:50.060516 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.060496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:56:50.064237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.064215    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-228a5f03-openshift-default-5d5dd8d768-bkdcc"]
Sep 16 00:56:50.104645 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.104620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ddx\" (UniqueName: \"kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx\") pod \"test-curl-9361-819dfa\" (UID: \"7baa3bf7-ac9c-4801-91e6-05875ad4df64\") " pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:50.113971 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.113924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ddx\" (UniqueName: \"kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx\") pod \"test-curl-9361-819dfa\" (UID: \"7baa3bf7-ac9c-4801-91e6-05875ad4df64\") " pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:50.173132 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.173100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:50.336201 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.336165    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" path="/var/lib/kubelet/pods/0e84981f-1ed5-409e-ab37-670c43190ca1/volumes"
Sep 16 00:56:50.447184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:50.447152    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9361-819dfa"]
Sep 16 00:56:50.449401 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:56:50.449373    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7baa3bf7_ac9c_4801_91e6_05875ad4df64.slice/crio-d35776886849b92b3b7a034a6b6fedccd512d179cc7991c0fb9ed83cc9e5dffe WatchSource:0}: Error finding container d35776886849b92b3b7a034a6b6fedccd512d179cc7991c0fb9ed83cc9e5dffe: Status 404 returned error can't find the container with id d35776886849b92b3b7a034a6b6fedccd512d179cc7991c0fb9ed83cc9e5dffe
Sep 16 00:56:51.045791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:51.045750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9361-819dfa" event={"ID":"7baa3bf7-ac9c-4801-91e6-05875ad4df64","Type":"ContainerStarted","Data":"fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602"}
Sep 16 00:56:51.045791 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:51.045797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9361-819dfa" event={"ID":"7baa3bf7-ac9c-4801-91e6-05875ad4df64","Type":"ContainerStarted","Data":"d35776886849b92b3b7a034a6b6fedccd512d179cc7991c0fb9ed83cc9e5dffe"}
Sep 16 00:56:51.070758 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:51.070704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9361-819dfa" podStartSLOduration=2.070687755 podStartE2EDuration="2.070687755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:56:51.069122592 +0000 UTC m=+1381.230632864" watchObservedRunningTime="2026-09-16 00:56:51.070687755 +0000 UTC m=+1381.232198027"
Sep 16 00:56:51.683590 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:51.683547    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9361-819dfa"]
Sep 16 00:56:51.689911 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:51.689875    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9361-819dfa"]
Sep 16 00:56:52.058067 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:56:52.058035    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9361-819dfa" container="test-curl-9361-819dfa"
Sep 16 00:56:53.063845 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:53.063785    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9361-819dfa" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" containerName="test-curl-9361-819dfa" containerID="cri-o://fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602" gracePeriod=2
Sep 16 00:56:55.245211 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:55.245185    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:55.352184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:55.352089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95ddx\" (UniqueName: \"kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx\") pod \"7baa3bf7-ac9c-4801-91e6-05875ad4df64\" (UID: \"7baa3bf7-ac9c-4801-91e6-05875ad4df64\") "
Sep 16 00:56:55.354453 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:55.354419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx" (OuterVolumeSpecName: "kube-api-access-95ddx") pod "7baa3bf7-ac9c-4801-91e6-05875ad4df64" (UID: "7baa3bf7-ac9c-4801-91e6-05875ad4df64"). InnerVolumeSpecName "kube-api-access-95ddx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:56:55.453794 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:55.453758    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95ddx\" (UniqueName: \"kubernetes.io/projected/7baa3bf7-ac9c-4801-91e6-05875ad4df64-kube-api-access-95ddx\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:56:56.082574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.082542    2577 generic.go:356] "Generic (PLEG): container finished" podID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" containerID="fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602" exitCode=137
Sep 16 00:56:56.082574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.082577    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:56.082813 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.082636    2577 scope.go:117] "RemoveContainer" containerID="fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602"
Sep 16 00:56:56.092549 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.092533    2577 scope.go:117] "RemoveContainer" containerID="fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602"
Sep 16 00:56:56.092890 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:56:56.092867    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602\": container with ID starting with fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602 not found: ID does not exist" containerID="fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602"
Sep 16 00:56:56.092968 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.092903    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602"} err="failed to get container status \"fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602\": rpc error: code = NotFound desc = could not find container \"fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602\": container with ID starting with fa8295a49d190bcdcd82efab9b9a3e95977d2e673f77879a0f78fe6848acf602 not found: ID does not exist"
Sep 16 00:56:56.092968 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:56:56.092875    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9361-819dfa\" is forbidden: User \"system:node:ip-10-0-134-152.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-152.ec2.internal' and this object" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" pod="opendatahub/test-curl-9361-819dfa"
Sep 16 00:56:56.334407 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:56.334328    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" path="/var/lib/kubelet/pods/7baa3bf7-ac9c-4801-91e6-05875ad4df64/volumes"
Sep 16 00:56:58.804788 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.804751    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 00:56:58.805552 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.805518    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:58.805552 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.805547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e84981f-1ed5-409e-ab37-670c43190ca1" containerName="istio-proxy"
Sep 16 00:56:58.805746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.805586    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" containerName="test-curl-9361-819dfa"
Sep 16 00:56:58.805746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.805595    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" containerName="test-curl-9361-819dfa"
Sep 16 00:56:58.805746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.805728    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7baa3bf7-ac9c-4801-91e6-05875ad4df64" containerName="test-curl-9361-819dfa"
Sep 16 00:56:58.810884 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.810864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.813483 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.813460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3462ea-openshift-default-dockercfg-qswmn\""
Sep 16 00:56:58.819246 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.819220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 00:56:58.887685 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.887856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjjk\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.887856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.887856 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.888033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.888033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.888033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.887943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.888033 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.888025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.888191 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.888059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.988628 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.988805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.988805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.988805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjjk\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.988805 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989155 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.988927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989352 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.989303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989510 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.989484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989645 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.989622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.989741 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.989726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.990216 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.990191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.991327 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.991304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.991809 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.991784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.996734 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.996703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:58.996837 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:58.996794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjjk\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk\") pod \"e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") " pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:59.123501 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:59.123412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:56:59.402757 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:59.402725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 00:56:59.404855 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:56:59.404807    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1096bd0_3b30_4565_84d7_833f9b0f405d.slice/crio-4181bb612811eced7bf6b67c28786e2b5ab8f30d87e18db64bea93e559a160ae WatchSource:0}: Error finding container 4181bb612811eced7bf6b67c28786e2b5ab8f30d87e18db64bea93e559a160ae: Status 404 returned error can't find the container with id 4181bb612811eced7bf6b67c28786e2b5ab8f30d87e18db64bea93e559a160ae
Sep 16 00:56:59.407357 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:59.407321    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:59.407453 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:59.407396    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:56:59.407453 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:56:59.407429    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:57:00.107374 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:00.107339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerStarted","Data":"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"}
Sep 16 00:57:00.107793 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:00.107393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerStarted","Data":"4181bb612811eced7bf6b67c28786e2b5ab8f30d87e18db64bea93e559a160ae"}
Sep 16 00:57:00.123567 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:00.123540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:00.128786 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:00.128738    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podStartSLOduration=2.12872242 podStartE2EDuration="2.12872242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:00.127290183 +0000 UTC m=+1390.288800455" watchObservedRunningTime="2026-09-16 00:57:00.12872242 +0000 UTC m=+1390.290232691"
Sep 16 00:57:00.129846 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:00.129824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:01.113454 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:01.113407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:01.114688 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:01.114661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:03.405059 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.405023    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 00:57:03.409237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.409211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.411877 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.411854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3462ea-serving-cert\""
Sep 16 00:57:03.421267 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.421244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 00:57:03.533785 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.533749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.533987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.533813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.533987 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.533899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpj79\" (UniqueName: \"kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.534075 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.533983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.635354 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.635308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.635574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.635381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.635574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.635429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.635574 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.635499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpj79\" (UniqueName: \"kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.636111 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.636088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.637983 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.637939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.638078 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.638003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.643476 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.643448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpj79\" (UniqueName: \"kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79\") pod \"maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:03.725305 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:03.725218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:04.001475 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:04.001448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 00:57:04.003315 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:57:04.003286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c74a99_8124_47e9_a5e0_71e17f342925.slice/crio-febdc6fa9f67c31c378953588540187b94b5c0700c4d19ee635b89ce1824d7a2 WatchSource:0}: Error finding container febdc6fa9f67c31c378953588540187b94b5c0700c4d19ee635b89ce1824d7a2: Status 404 returned error can't find the container with id febdc6fa9f67c31c378953588540187b94b5c0700c4d19ee635b89ce1824d7a2
Sep 16 00:57:04.134485 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:04.134452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" event={"ID":"31c74a99-8124-47e9-a5e0-71e17f342925","Type":"ContainerStarted","Data":"febdc6fa9f67c31c378953588540187b94b5c0700c4d19ee635b89ce1824d7a2"}
Sep 16 00:57:05.141900 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:05.141859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" event={"ID":"31c74a99-8124-47e9-a5e0-71e17f342925","Type":"ContainerStarted","Data":"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"}
Sep 16 00:57:05.142350 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:05.141938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:05.159382 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:05.159327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" podStartSLOduration=1.766680987 podStartE2EDuration="2.159310512s" totalImagesPullingTime="392.629525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:05.157210596 +0000 UTC m=+1395.318720868" watchObservedRunningTime="2026-09-16 00:57:05.159310512 +0000 UTC m=+1395.320820784"
Sep 16 00:57:05.327863 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:05.327825    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:57:11.165254 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:11.165214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 00:57:17.155123 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:17.155078    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 00:57:17.155523 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:17.155150    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 00:57:17.328441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:17.328403    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:57:19.222565 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.222467    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerID="3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf" exitCode=0
Sep 16 00:57:19.222565 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.222548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerDied","Data":"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"}
Sep 16 00:57:19.223115 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.223040    2577 scope.go:117] "RemoveContainer" containerID="3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"
Sep 16 00:57:19.224253 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.224225    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:57:19.224328 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.224294    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:57:19.224365 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:19.224326    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 00:57:20.230976 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:20.230907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerStarted","Data":"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"}
Sep 16 00:57:21.124647 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:21.124606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:21.124838 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:21.124795    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 16 00:57:21.124883 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:21.124855    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 16 00:57:22.124332 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:22.124295    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 16 00:57:22.124741 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:22.124393    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 16 00:57:23.124046 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:23.124002    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 16 00:57:23.124248 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:23.124068    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 16 00:57:24.123932 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:24.123895    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 16 00:57:24.124355 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:24.123977    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 16 00:57:25.123962 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:25.123910    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 16 00:57:25.124393 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:25.123985    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 16 00:57:26.127615 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:26.127585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:26.128101 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:26.128078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:26.128686 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:26.128667    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 00:57:32.794273 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.794234    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 00:57:32.798997 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.798973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.801714 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.801669    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-409cde-serving-cert\""
Sep 16 00:57:32.808106 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.808078    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 00:57:32.842101 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.842063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.842269 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.842116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.842269 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.842186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.842269 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.842254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvv4\" (UniqueName: \"kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.943299 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.943267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.943515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.943328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.943515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.943391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.943515 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.943424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvv4\" (UniqueName: \"kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.943692 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:57:32.943521    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-serving-cert: secret "maas-api-e2e-shared-b-w1-409cde-serving-cert" not found
Sep 16 00:57:32.943692 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:57:32.943597    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls podName:ef4e0c6e-e721-4da3-b706-3493b2174e2b nodeName:}" failed. No retries permitted until 2026-09-16 00:57:33.443575513 +0000 UTC m=+1423.605085779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls") pod "maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b") : secret "maas-api-e2e-shared-b-w1-409cde-serving-cert" not found
Sep 16 00:57:32.944168 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.944110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.946041 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.946021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:32.951612 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:32.951591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvv4\" (UniqueName: \"kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:33.448201 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:33.448159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:33.448385 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:57:33.448297    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-serving-cert: secret "maas-api-e2e-shared-b-w1-409cde-serving-cert" not found
Sep 16 00:57:33.448385 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:57:33.448377    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls podName:ef4e0c6e-e721-4da3-b706-3493b2174e2b nodeName:}" failed. No retries permitted until 2026-09-16 00:57:34.448360491 +0000 UTC m=+1424.609870741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls") pod "maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b") : secret "maas-api-e2e-shared-b-w1-409cde-serving-cert" not found
Sep 16 00:57:34.459215 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:34.459175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:34.461796 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:34.461771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:34.615650 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:34.615616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:34.895184 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:34.895156    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 00:57:34.897165 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:57:34.897129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4e0c6e_e721_4da3_b706_3493b2174e2b.slice/crio-524b62a11e31f1ae4c1b8d4a5ec3a581c9df7805e57c352fed1f8eecbcfa6e52 WatchSource:0}: Error finding container 524b62a11e31f1ae4c1b8d4a5ec3a581c9df7805e57c352fed1f8eecbcfa6e52: Status 404 returned error can't find the container with id 524b62a11e31f1ae4c1b8d4a5ec3a581c9df7805e57c352fed1f8eecbcfa6e52
Sep 16 00:57:35.328567 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:35.328530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" event={"ID":"ef4e0c6e-e721-4da3-b706-3493b2174e2b","Type":"ContainerStarted","Data":"524b62a11e31f1ae4c1b8d4a5ec3a581c9df7805e57c352fed1f8eecbcfa6e52"}
Sep 16 00:57:36.337066 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:36.337028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" event={"ID":"ef4e0c6e-e721-4da3-b706-3493b2174e2b","Type":"ContainerStarted","Data":"e2bbcad06b3b0d9c13b0a3e633b84da89cfd009d5ca902a65fd8b81a92f4a1d2"}
Sep 16 00:57:36.337488 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:36.337164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:57:36.355282 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:36.355231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" podStartSLOduration=3.716066976 podStartE2EDuration="4.355215485s" totalImagesPullingTime="639.148509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:57:36.353249875 +0000 UTC m=+1426.514760160" watchObservedRunningTime="2026-09-16 00:57:36.355215485 +0000 UTC m=+1426.516725757"
Sep 16 00:57:42.351596 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:57:42.351520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 00:58:04.510642 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.510592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9358-d3800b"]
Sep 16 00:58:04.517011 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.516981    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9358-d3800b"]
Sep 16 00:58:04.517180 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.517042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:04.519576 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.519553    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qndhv\""
Sep 16 00:58:04.655804 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.655771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkpk\" (UniqueName: \"kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk\") pod \"test-curl-9358-d3800b\" (UID: \"fee96f32-2df8-46ca-81ad-2008c58948df\") " pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:04.757237 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.757201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkpk\" (UniqueName: \"kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk\") pod \"test-curl-9358-d3800b\" (UID: \"fee96f32-2df8-46ca-81ad-2008c58948df\") " pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:04.766083 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.766017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkpk\" (UniqueName: \"kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk\") pod \"test-curl-9358-d3800b\" (UID: \"fee96f32-2df8-46ca-81ad-2008c58948df\") " pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:04.827081 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:04.827051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:05.097503 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:05.097465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9358-d3800b"]
Sep 16 00:58:05.099497 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:58:05.099467    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee96f32_2df8_46ca_81ad_2008c58948df.slice/crio-38eed5faf87f49b60feb77b4bae99f4768e547eb3ebd5460d3bd2198d78cc6a9 WatchSource:0}: Error finding container 38eed5faf87f49b60feb77b4bae99f4768e547eb3ebd5460d3bd2198d78cc6a9: Status 404 returned error can't find the container with id 38eed5faf87f49b60feb77b4bae99f4768e547eb3ebd5460d3bd2198d78cc6a9
Sep 16 00:58:05.504689 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:05.504661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9358-d3800b" event={"ID":"fee96f32-2df8-46ca-81ad-2008c58948df","Type":"ContainerStarted","Data":"b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c"}
Sep 16 00:58:05.504874 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:05.504703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9358-d3800b" event={"ID":"fee96f32-2df8-46ca-81ad-2008c58948df","Type":"ContainerStarted","Data":"38eed5faf87f49b60feb77b4bae99f4768e547eb3ebd5460d3bd2198d78cc6a9"}
Sep 16 00:58:05.520426 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:05.520378    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9358-d3800b" podStartSLOduration=1.5203633829999998 podStartE2EDuration="1.520363383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:05.518705219 +0000 UTC m=+1455.680215490" watchObservedRunningTime="2026-09-16 00:58:05.520363383 +0000 UTC m=+1455.681873653"
Sep 16 00:58:06.132096 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:06.132066    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9358-d3800b"]
Sep 16 00:58:06.135810 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:06.135778    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9358-d3800b"]
Sep 16 00:58:06.511269 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:58:06.511237    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9358-d3800b" container="test-curl-9358-d3800b"
Sep 16 00:58:07.515939 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:07.515897    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9358-d3800b" podUID="fee96f32-2df8-46ca-81ad-2008c58948df" containerName="test-curl-9358-d3800b" containerID="cri-o://b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c" gracePeriod=2
Sep 16 00:58:08.328372 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:08.328327    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:58:09.699872 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:09.699844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:09.814409 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:09.814317    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxkpk\" (UniqueName: \"kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk\") pod \"fee96f32-2df8-46ca-81ad-2008c58948df\" (UID: \"fee96f32-2df8-46ca-81ad-2008c58948df\") "
Sep 16 00:58:09.816746 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:09.816697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk" (OuterVolumeSpecName: "kube-api-access-pxkpk") pod "fee96f32-2df8-46ca-81ad-2008c58948df" (UID: "fee96f32-2df8-46ca-81ad-2008c58948df"). InnerVolumeSpecName "kube-api-access-pxkpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 00:58:09.915725 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:09.915679    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkpk\" (UniqueName: \"kubernetes.io/projected/fee96f32-2df8-46ca-81ad-2008c58948df-kube-api-access-pxkpk\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 00:58:10.335094 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.335059    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee96f32-2df8-46ca-81ad-2008c58948df" path="/var/lib/kubelet/pods/fee96f32-2df8-46ca-81ad-2008c58948df/volumes"
Sep 16 00:58:10.537683 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.537639    2577 generic.go:356] "Generic (PLEG): container finished" podID="fee96f32-2df8-46ca-81ad-2008c58948df" containerID="b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c" exitCode=137
Sep 16 00:58:10.537879 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.537701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9358-d3800b"
Sep 16 00:58:10.537879 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.537716    2577 scope.go:117] "RemoveContainer" containerID="b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c"
Sep 16 00:58:10.547655 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.547634    2577 scope.go:117] "RemoveContainer" containerID="b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c"
Sep 16 00:58:10.548011 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:58:10.547992    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c\": container with ID starting with b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c not found: ID does not exist" containerID="b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c"
Sep 16 00:58:10.548093 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:10.548020    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c"} err="failed to get container status \"b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c\": rpc error: code = NotFound desc = could not find container \"b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c\": container with ID starting with b543e81fe9276c7475bd3c622f82119a8e38684c8ba0446ceb3dd4533ad2186c not found: ID does not exist"
Sep 16 00:58:12.649040 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.649006    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 00:58:12.649486 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.649472    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee96f32-2df8-46ca-81ad-2008c58948df" containerName="test-curl-9358-d3800b"
Sep 16 00:58:12.649486 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.649487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee96f32-2df8-46ca-81ad-2008c58948df" containerName="test-curl-9358-d3800b"
Sep 16 00:58:12.649571 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.649553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee96f32-2df8-46ca-81ad-2008c58948df" containerName="test-curl-9358-d3800b"
Sep 16 00:58:12.654157 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.654139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.656626 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.656596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-37b46587-serving-cert\""
Sep 16 00:58:12.671999 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.661962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 00:58:12.738478 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.738440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.738649 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.738552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.738649 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.738604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zb9x\" (UniqueName: \"kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.738725 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.738668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.839355 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.839318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.839561 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.839371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.839561 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.839446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.839561 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.839485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zb9x\" (UniqueName: \"kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.839561 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:58:12.839555    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-serving-cert: secret "maas-api-e2e-mt-37b46587-serving-cert" not found
Sep 16 00:58:12.839794 ip-10-0-134-152 kubenswrapper[2577]: E0916 00:58:12.839635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls podName:f6ace32f-4454-4739-aee5-0e9363f01633 nodeName:}" failed. No retries permitted until 2026-09-16 00:58:13.339614829 +0000 UTC m=+1463.501125086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls") pod "maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" (UID: "f6ace32f-4454-4739-aee5-0e9363f01633") : secret "maas-api-e2e-mt-37b46587-serving-cert" not found
Sep 16 00:58:12.840115 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.840094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.842293 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.842267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:12.850552 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:12.850526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zb9x\" (UniqueName: \"kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:13.343274 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:13.343242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:13.345826 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:13.345803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") pod \"maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:13.580071 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:13.580040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:13.854277 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:13.854251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 00:58:13.856761 ip-10-0-134-152 kubenswrapper[2577]: W0916 00:58:13.856729    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ace32f_4454_4739_aee5_0e9363f01633.slice/crio-54da7666ba4b484cd417578272442caaa9259b89472f87b6d89ef7ff5b667cf1 WatchSource:0}: Error finding container 54da7666ba4b484cd417578272442caaa9259b89472f87b6d89ef7ff5b667cf1: Status 404 returned error can't find the container with id 54da7666ba4b484cd417578272442caaa9259b89472f87b6d89ef7ff5b667cf1
Sep 16 00:58:14.564566 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:14.564521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" event={"ID":"f6ace32f-4454-4739-aee5-0e9363f01633","Type":"ContainerStarted","Data":"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"}
Sep 16 00:58:14.564743 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:14.564589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" event={"ID":"f6ace32f-4454-4739-aee5-0e9363f01633","Type":"ContainerStarted","Data":"54da7666ba4b484cd417578272442caaa9259b89472f87b6d89ef7ff5b667cf1"}
Sep 16 00:58:14.564743 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:14.564713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:14.582623 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:14.582572    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" podStartSLOduration=2.065054109 podStartE2EDuration="2.582554591s" totalImagesPullingTime="517.500482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 00:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 00:58:14.581363782 +0000 UTC m=+1464.742874053" watchObservedRunningTime="2026-09-16 00:58:14.582554591 +0000 UTC m=+1464.744064864"
Sep 16 00:58:20.575375 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:20.575343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 00:58:44.328249 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:44.328208    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 00:58:50.422441 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:50.422414    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:58:50.429972 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:50.429932    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 00:58:50.572858 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:58:50.572819    2577 scope.go:117] "RemoveContainer" containerID="a081cfc9e885f4e8f2b70d41e67c7fadcc8dab57e75be7387818a5d7000fb17f"
Sep 16 00:59:09.327464 ip-10-0-134-152 kubenswrapper[2577]: I0916 00:59:09.327428    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:00:01.263006 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.262942    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 01:00:01.263559 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.263214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" podUID="f6ace32f-4454-4739-aee5-0e9363f01633" containerName="maas-api" containerID="cri-o://f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144" gracePeriod=30
Sep 16 01:00:01.536161 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.536131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 01:00:01.546275 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.546241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs\") pod \"f6ace32f-4454-4739-aee5-0e9363f01633\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") "
Sep 16 01:00:01.546275 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.546273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") pod \"f6ace32f-4454-4739-aee5-0e9363f01633\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") "
Sep 16 01:00:01.546520 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.546311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca\") pod \"f6ace32f-4454-4739-aee5-0e9363f01633\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") "
Sep 16 01:00:01.546520 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.546331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zb9x\" (UniqueName: \"kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x\") pod \"f6ace32f-4454-4739-aee5-0e9363f01633\" (UID: \"f6ace32f-4454-4739-aee5-0e9363f01633\") "
Sep 16 01:00:01.546739 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.546702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6ace32f-4454-4739-aee5-0e9363f01633" (UID: "f6ace32f-4454-4739-aee5-0e9363f01633"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:00:01.547067 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.547033    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ace32f-4454-4739-aee5-0e9363f01633-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:00:01.549250 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.549171    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x" (OuterVolumeSpecName: "kube-api-access-2zb9x") pod "f6ace32f-4454-4739-aee5-0e9363f01633" (UID: "f6ace32f-4454-4739-aee5-0e9363f01633"). InnerVolumeSpecName "kube-api-access-2zb9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:00:01.549583 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.549557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6ace32f-4454-4739-aee5-0e9363f01633" (UID: "f6ace32f-4454-4739-aee5-0e9363f01633"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:00:01.549668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.549580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6ace32f-4454-4739-aee5-0e9363f01633" (UID: "f6ace32f-4454-4739-aee5-0e9363f01633"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:00:01.648200 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.648163    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:00:01.648200 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.648195    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ace32f-4454-4739-aee5-0e9363f01633-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:00:01.648200 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:01.648204    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zb9x\" (UniqueName: \"kubernetes.io/projected/f6ace32f-4454-4739-aee5-0e9363f01633-kube-api-access-2zb9x\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:00:02.173613 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.173577    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6ace32f-4454-4739-aee5-0e9363f01633" containerID="f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144" exitCode=0
Sep 16 01:00:02.173800 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.173620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"
Sep 16 01:00:02.173800 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.173662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" event={"ID":"f6ace32f-4454-4739-aee5-0e9363f01633","Type":"ContainerDied","Data":"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"}
Sep 16 01:00:02.173800 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.173717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq" event={"ID":"f6ace32f-4454-4739-aee5-0e9363f01633","Type":"ContainerDied","Data":"54da7666ba4b484cd417578272442caaa9259b89472f87b6d89ef7ff5b667cf1"}
Sep 16 01:00:02.173800 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.173741    2577 scope.go:117] "RemoveContainer" containerID="f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"
Sep 16 01:00:02.184106 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.184084    2577 scope.go:117] "RemoveContainer" containerID="f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"
Sep 16 01:00:02.184391 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:00:02.184373    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144\": container with ID starting with f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144 not found: ID does not exist" containerID="f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"
Sep 16 01:00:02.184444 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.184401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144"} err="failed to get container status \"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144\": rpc error: code = NotFound desc = could not find container \"f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144\": container with ID starting with f38f88cad0ef6eb9306c474bbf95eea1221ea880ee2cf04f7d13f919032c1144 not found: ID does not exist"
Sep 16 01:00:02.196674 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.196651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 01:00:02.206940 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.206918    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-37b46587-7db9759b9f-8zmnq"]
Sep 16 01:00:02.334111 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:02.334076    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ace32f-4454-4739-aee5-0e9363f01633" path="/var/lib/kubelet/pods/f6ace32f-4454-4739-aee5-0e9363f01633/volumes"
Sep 16 01:00:11.327928 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:11.327886    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:00:30.331981 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:30.331924    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:00:49.877114 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.877015    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:00:49.877732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.877532    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ace32f-4454-4739-aee5-0e9363f01633" containerName="maas-api"
Sep 16 01:00:49.877732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.877545    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ace32f-4454-4739-aee5-0e9363f01633" containerName="maas-api"
Sep 16 01:00:49.877732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.877643    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ace32f-4454-4739-aee5-0e9363f01633" containerName="maas-api"
Sep 16 01:00:49.880868 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.880848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.885595 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.885572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-715834-openshift-default-dockercfg-snbpw\""
Sep 16 01:00:49.901322 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.901289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:00:49.934517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.934813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5jg\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.935102 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:49.935102 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:49.934903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.035812 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.035774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.035812 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.035820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.035855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.035888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.035915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5jg\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036801 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036866 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.036973 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.036932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.038907 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.038879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.039025 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.038970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.050412 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.050382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5jg\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.053432 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.053396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token\") pod \"e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") " pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.193128 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.193031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:50.474761 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.474728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:00:50.476046 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:00:50.476014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cf260b_293a_4718_88f2_f8790dc97434.slice/crio-851b024de40f5626b043f0891089c45c5ecc9db11b5dc5f1ba4494f16321bc59 WatchSource:0}: Error finding container 851b024de40f5626b043f0891089c45c5ecc9db11b5dc5f1ba4494f16321bc59: Status 404 returned error can't find the container with id 851b024de40f5626b043f0891089c45c5ecc9db11b5dc5f1ba4494f16321bc59
Sep 16 01:00:50.478431 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.478396    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:50.478491 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.478466    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:50.478540 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:50.478496    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:00:51.449574 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:51.449531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerStarted","Data":"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"}
Sep 16 01:00:51.449990 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:51.449587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerStarted","Data":"851b024de40f5626b043f0891089c45c5ecc9db11b5dc5f1ba4494f16321bc59"}
Sep 16 01:00:51.470499 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:51.470449    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podStartSLOduration=2.4704339920000002 podStartE2EDuration="2.470433992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:51.468047815 +0000 UTC m=+1621.629558083" watchObservedRunningTime="2026-09-16 01:00:51.470433992 +0000 UTC m=+1621.631944275"
Sep 16 01:00:52.193383 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:52.193349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:52.198556 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:52.198528    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:52.456479 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:52.456392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:52.457651 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:52.457626    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:00:57.031686 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.031647    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:00:57.037662 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.037636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.040584 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.040558    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-715834-serving-cert\""
Sep 16 01:00:57.051374 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.051329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:00:57.111710 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.111664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.111915 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.111722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.111915 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.111814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.111915 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.111906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fjg\" (UniqueName: \"kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.213478 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.213429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.213681 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.213490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.213681 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.213537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.213681 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:00:57.213598    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-serving-cert: secret "maas-api-e2e-ipp-a-715834-serving-cert" not found
Sep 16 01:00:57.213681 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.213626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fjg\" (UniqueName: \"kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.213681 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:00:57.213671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls podName:9c92a798-f9be-4f5a-b309-5e10ff9cebf9 nodeName:}" failed. No retries permitted until 2026-09-16 01:00:57.713649255 +0000 UTC m=+1627.875159505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls") pod "maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" (UID: "9c92a798-f9be-4f5a-b309-5e10ff9cebf9") : secret "maas-api-e2e-ipp-a-715834-serving-cert" not found
Sep 16 01:00:57.214366 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.214341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.216264 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.216242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.222420 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.222393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fjg\" (UniqueName: \"kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.719847 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.719802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.722517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.722497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:57.953104 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:57.953071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:58.233609 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:58.233566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:00:58.235667 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:00:58.235644    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c92a798_f9be_4f5a_b309_5e10ff9cebf9.slice/crio-192dc659cf5d8aca7a172ff65ea7ecda13ada0c1010918e54409fa67eb99b109 WatchSource:0}: Error finding container 192dc659cf5d8aca7a172ff65ea7ecda13ada0c1010918e54409fa67eb99b109: Status 404 returned error can't find the container with id 192dc659cf5d8aca7a172ff65ea7ecda13ada0c1010918e54409fa67eb99b109
Sep 16 01:00:58.237128 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:58.237104    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:00:58.503007 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:58.502901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" event={"ID":"9c92a798-f9be-4f5a-b309-5e10ff9cebf9","Type":"ContainerStarted","Data":"192dc659cf5d8aca7a172ff65ea7ecda13ada0c1010918e54409fa67eb99b109"}
Sep 16 01:00:59.511566 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:59.511523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" event={"ID":"9c92a798-f9be-4f5a-b309-5e10ff9cebf9","Type":"ContainerStarted","Data":"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"}
Sep 16 01:00:59.512098 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:59.511731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:00:59.531664 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:00:59.531605    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" podStartSLOduration=2.19158892 podStartE2EDuration="2.53158903s" totalImagesPullingTime="340.00011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:00:59.531129753 +0000 UTC m=+1629.692640026" watchObservedRunningTime="2026-09-16 01:00:59.53158903 +0000 UTC m=+1629.693099303"
Sep 16 01:01:05.523008 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:05.522965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:01:11.593588 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.593552    2577 generic.go:356] "Generic (PLEG): container finished" podID="52cf260b-293a-4718-88f2-f8790dc97434" containerID="0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2" exitCode=0
Sep 16 01:01:11.594026 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.593621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerDied","Data":"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"}
Sep 16 01:01:11.594074 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.594064    2577 scope.go:117] "RemoveContainer" containerID="0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"
Sep 16 01:01:11.595191 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.595165    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:11.595275 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.595234    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:11.595275 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:11.595268    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 01:01:12.601652 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:12.601617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerStarted","Data":"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"}
Sep 16 01:01:13.193331 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:13.193293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:01:13.193577 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:13.193551    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 16 01:01:13.193633 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:13.193606    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 16 01:01:14.193387 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:14.193347    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 16 01:01:14.193802 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:14.193414    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 16 01:01:15.193871 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:15.193830    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 16 01:01:15.194320 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:15.193904    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 16 01:01:15.327807 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:15.327764    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:01:16.194272 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:16.194235    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 16 01:01:16.194713 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:16.194294    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 16 01:01:17.193481 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:17.193433    2577 patch_prober.go:28] interesting pod/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 16 01:01:17.193681 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:17.193501    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 16 01:01:18.197417 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:18.197383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:01:18.197891 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:18.197685    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:01:18.198519 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:18.198501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:01:30.613898 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.613854    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:01:30.617329 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.617305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.620145 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.620116    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0e8a7b-serving-cert\""
Sep 16 01:01:30.631408 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.631380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:01:30.661403 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.661360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.661618 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.661410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.661618 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.661517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6s8\" (UniqueName: \"kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.661618 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.661546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.763128 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.763085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6s8\" (UniqueName: \"kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.763367 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.763154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.763367 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.763221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.763367 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.763251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.763367 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:01:30.763332    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-serving-cert: secret "maas-api-e2e-ipp-b-0e8a7b-serving-cert" not found
Sep 16 01:01:30.763610 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:01:30.763430    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls podName:f18a5872-a3ec-43ee-be18-f12541ecfcbf nodeName:}" failed. No retries permitted until 2026-09-16 01:01:31.263406658 +0000 UTC m=+1661.424916910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls") pod "maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" (UID: "f18a5872-a3ec-43ee-be18-f12541ecfcbf") : secret "maas-api-e2e-ipp-b-0e8a7b-serving-cert" not found
Sep 16 01:01:30.764229 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.764182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.766008 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.765986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:30.772390 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:30.772366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6s8\" (UniqueName: \"kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:31.267565 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:31.267527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:31.270485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:31.270448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:31.532919 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:31.532809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:31.808157 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:31.808125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:01:31.809585 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:01:31.809546    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18a5872_a3ec_43ee_be18_f12541ecfcbf.slice/crio-00cc6f87dc6c640294051154a80cd1322f54cadf1b5dfdbbebf21d92ed73f933 WatchSource:0}: Error finding container 00cc6f87dc6c640294051154a80cd1322f54cadf1b5dfdbbebf21d92ed73f933: Status 404 returned error can't find the container with id 00cc6f87dc6c640294051154a80cd1322f54cadf1b5dfdbbebf21d92ed73f933
Sep 16 01:01:32.713790 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:32.713751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" event={"ID":"f18a5872-a3ec-43ee-be18-f12541ecfcbf","Type":"ContainerStarted","Data":"00cc6f87dc6c640294051154a80cd1322f54cadf1b5dfdbbebf21d92ed73f933"}
Sep 16 01:01:33.721294 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:33.721248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" event={"ID":"f18a5872-a3ec-43ee-be18-f12541ecfcbf","Type":"ContainerStarted","Data":"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"}
Sep 16 01:01:33.721754 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:33.721384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:33.740924 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:33.740864    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" podStartSLOduration=2.787802662 podStartE2EDuration="3.74084465s" totalImagesPullingTime="953.041988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:33.738978077 +0000 UTC m=+1663.900488353" watchObservedRunningTime="2026-09-16 01:01:33.74084465 +0000 UTC m=+1663.902354975"
Sep 16 01:01:39.732156 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:39.732124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:01:46.327764 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:46.327727    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:01:56.594443 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.594388    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:01:56.599439 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.599408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.602261 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.602235    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2c31c5-serving-cert\""
Sep 16 01:01:56.608754 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.608726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:01:56.631121 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.631077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pqt\" (UniqueName: \"kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.631294 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.631136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.631294 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.631199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.631294 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.631256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.732723 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.732679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.732971 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.732844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pqt\" (UniqueName: \"kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.732971 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.732888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.732971 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.732919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.733177 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:01:56.733028    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-serving-cert: secret "maas-api-e2e-api-a-2c31c5-serving-cert" not found
Sep 16 01:01:56.733177 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:01:56.733107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls podName:25211d3a-5e80-442a-8415-1d097007d5d5 nodeName:}" failed. No retries permitted until 2026-09-16 01:01:57.23308556 +0000 UTC m=+1687.394595815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls") pod "maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" (UID: "25211d3a-5e80-442a-8415-1d097007d5d5") : secret "maas-api-e2e-api-a-2c31c5-serving-cert" not found
Sep 16 01:01:56.733617 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.733597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.735535 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.735508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:56.741524 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:56.741503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pqt\" (UniqueName: \"kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:57.239619 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.239587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:57.242316 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.242290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") pod \"maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:57.514119 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.514077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:57.691234 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.691202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:01:57.797137 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.797105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:01:57.799406 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:01:57.799380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25211d3a_5e80_442a_8415_1d097007d5d5.slice/crio-be864b895dd3bd77fb3a33eb3e01cf2a8fb25a7da7c660b77a3df26e228043e1 WatchSource:0}: Error finding container be864b895dd3bd77fb3a33eb3e01cf2a8fb25a7da7c660b77a3df26e228043e1: Status 404 returned error can't find the container with id be864b895dd3bd77fb3a33eb3e01cf2a8fb25a7da7c660b77a3df26e228043e1
Sep 16 01:01:57.834259 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.834225    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:01:57.873780 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:57.873741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" event={"ID":"25211d3a-5e80-442a-8415-1d097007d5d5","Type":"ContainerStarted","Data":"be864b895dd3bd77fb3a33eb3e01cf2a8fb25a7da7c660b77a3df26e228043e1"}
Sep 16 01:01:59.886663 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:59.886623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" event={"ID":"25211d3a-5e80-442a-8415-1d097007d5d5","Type":"ContainerStarted","Data":"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"}
Sep 16 01:01:59.887224 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:59.886688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:01:59.905718 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:01:59.905666    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" podStartSLOduration=2.693924547 podStartE2EDuration="3.905650152s" totalImagesPullingTime="1.211725605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:01:59.902804968 +0000 UTC m=+1690.064315240" watchObservedRunningTime="2026-09-16 01:01:59.905650152 +0000 UTC m=+1690.067160423"
Sep 16 01:02:05.898397 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:05.898364    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:02:26.328168 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:26.328077    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:02:30.710850 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.710798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:02:30.715916 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.715889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.718626 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.718591    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-761551-serving-cert\""
Sep 16 01:02:30.732457 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.731683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:02:30.786548 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.786512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.786741 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.786568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.786741 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.786712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2h8\" (UniqueName: \"kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.786858 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.786830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.888321 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.888270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2h8\" (UniqueName: \"kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.888547 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.888385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.888547 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.888420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.888547 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.888463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.888547 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:02:30.888535    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-761551-serving-cert: secret "maas-api-e2e-api-b-761551-serving-cert" not found
Sep 16 01:02:30.888778 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:02:30.888622    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls podName:1e482e96-3bfc-4ef3-8647-ae6dd3522634 nodeName:}" failed. No retries permitted until 2026-09-16 01:02:31.388601835 +0000 UTC m=+1721.550112099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls") pod "maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" (UID: "1e482e96-3bfc-4ef3-8647-ae6dd3522634") : secret "maas-api-e2e-api-b-761551-serving-cert" not found
Sep 16 01:02:30.889358 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.889336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.891458 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.891428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:30.897243 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:30.897212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2h8\" (UniqueName: \"kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:31.394511 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:31.394463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:31.397095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:31.397066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") pod \"maas-api-e2e-api-b-761551-6c8c56856d-6rc5n\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:31.636604 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:31.636557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:31.915398 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:31.915371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:02:31.917196 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:02:31.917165    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e482e96_3bfc_4ef3_8647_ae6dd3522634.slice/crio-25a8e6c20aeaaf70c13fe587340d687f0272583e7ba0a5cf2bc218f7991f4086 WatchSource:0}: Error finding container 25a8e6c20aeaaf70c13fe587340d687f0272583e7ba0a5cf2bc218f7991f4086: Status 404 returned error can't find the container with id 25a8e6c20aeaaf70c13fe587340d687f0272583e7ba0a5cf2bc218f7991f4086
Sep 16 01:02:32.084174 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:32.084135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" event={"ID":"1e482e96-3bfc-4ef3-8647-ae6dd3522634","Type":"ContainerStarted","Data":"25a8e6c20aeaaf70c13fe587340d687f0272583e7ba0a5cf2bc218f7991f4086"}
Sep 16 01:02:34.098145 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:34.098104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" event={"ID":"1e482e96-3bfc-4ef3-8647-ae6dd3522634","Type":"ContainerStarted","Data":"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"}
Sep 16 01:02:34.098568 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:34.098191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:34.116392 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:34.116324    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" podStartSLOduration=2.987903164 podStartE2EDuration="4.116302274s" totalImagesPullingTime="1.12839911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:34.114466722 +0000 UTC m=+1724.275976995" watchObservedRunningTime="2026-09-16 01:02:34.116302274 +0000 UTC m=+1724.277812546"
Sep 16 01:02:40.110461 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:40.110428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:02:46.922448 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:46.922420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:02:47.049472 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:47.049442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:02:57.424708 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.424673    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:02:57.429111 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.429085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.432451 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.432228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-eba3d3-serving-cert\""
Sep 16 01:02:57.441367 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.441209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:02:57.469811 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.469768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2nq\" (UniqueName: \"kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.470007 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.469899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.470007 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.469974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.470104 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.470041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.571131 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.571097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.571320 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.571201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2nq\" (UniqueName: \"kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.571320 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.571259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.571414 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:02:57.571348    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-serving-cert: secret "maas-api-e2e-ipp-cleanup-eba3d3-serving-cert" not found
Sep 16 01:02:57.571414 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:02:57.571409    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls podName:ab1b31b4-4f0b-4234-b1c3-2d6acd60a798 nodeName:}" failed. No retries permitted until 2026-09-16 01:02:58.071389717 +0000 UTC m=+1748.232899982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" (UID: "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798") : secret "maas-api-e2e-ipp-cleanup-eba3d3-serving-cert" not found
Sep 16 01:02:57.571506 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.571437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.571849 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.571825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.574178 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.574154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:57.581889 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:57.581862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2nq\" (UniqueName: \"kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:58.077214 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:58.077179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:58.079910 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:58.079869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:58.328116 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:58.328018    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:02:58.346578 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:58.346547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:58.627502 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:58.627473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:02:58.629629 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:02:58.629600    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1b31b4_4f0b_4234_b1c3_2d6acd60a798.slice/crio-6b942d851185e848fc59d9d0be191f0c5014dd44f3a180f801e767264032476d WatchSource:0}: Error finding container 6b942d851185e848fc59d9d0be191f0c5014dd44f3a180f801e767264032476d: Status 404 returned error can't find the container with id 6b942d851185e848fc59d9d0be191f0c5014dd44f3a180f801e767264032476d
Sep 16 01:02:59.258269 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:59.258231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" event={"ID":"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798","Type":"ContainerStarted","Data":"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"}
Sep 16 01:02:59.258473 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:59.258299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" event={"ID":"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798","Type":"ContainerStarted","Data":"6b942d851185e848fc59d9d0be191f0c5014dd44f3a180f801e767264032476d"}
Sep 16 01:02:59.258473 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:59.258407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:02:59.279159 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:02:59.279099    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" podStartSLOduration=1.9120300989999999 podStartE2EDuration="2.279083161s" totalImagesPullingTime="367.053062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:02:59.276387602 +0000 UTC m=+1749.437897900" watchObservedRunningTime="2026-09-16 01:02:59.279083161 +0000 UTC m=+1749.440593434"
Sep 16 01:03:05.272461 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:05.272417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:03:19.276492 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.276453    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:03:19.277012 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.276772    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" podUID="25211d3a-5e80-442a-8415-1d097007d5d5" containerName="maas-api" containerID="cri-o://341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183" gracePeriod=30
Sep 16 01:03:19.546487 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.546451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:03:19.711202 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca\") pod \"25211d3a-5e80-442a-8415-1d097007d5d5\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") "
Sep 16 01:03:19.711413 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs\") pod \"25211d3a-5e80-442a-8415-1d097007d5d5\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") "
Sep 16 01:03:19.711413 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9pqt\" (UniqueName: \"kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt\") pod \"25211d3a-5e80-442a-8415-1d097007d5d5\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") "
Sep 16 01:03:19.711413 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") pod \"25211d3a-5e80-442a-8415-1d097007d5d5\" (UID: \"25211d3a-5e80-442a-8415-1d097007d5d5\") "
Sep 16 01:03:19.711874 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25211d3a-5e80-442a-8415-1d097007d5d5" (UID: "25211d3a-5e80-442a-8415-1d097007d5d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:03:19.711874 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.711739    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25211d3a-5e80-442a-8415-1d097007d5d5-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:19.714029 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.713992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25211d3a-5e80-442a-8415-1d097007d5d5" (UID: "25211d3a-5e80-442a-8415-1d097007d5d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:19.714480 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.714448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt" (OuterVolumeSpecName: "kube-api-access-h9pqt") pod "25211d3a-5e80-442a-8415-1d097007d5d5" (UID: "25211d3a-5e80-442a-8415-1d097007d5d5"). InnerVolumeSpecName "kube-api-access-h9pqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:19.714787 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.714761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25211d3a-5e80-442a-8415-1d097007d5d5" (UID: "25211d3a-5e80-442a-8415-1d097007d5d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:19.812751 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.812661    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:19.812751 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.812692    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9pqt\" (UniqueName: \"kubernetes.io/projected/25211d3a-5e80-442a-8415-1d097007d5d5-kube-api-access-h9pqt\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:19.812751 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:19.812701    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25211d3a-5e80-442a-8415-1d097007d5d5-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:20.394550 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.394512    2577 generic.go:356] "Generic (PLEG): container finished" podID="25211d3a-5e80-442a-8415-1d097007d5d5" containerID="341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183" exitCode=0
Sep 16 01:03:20.395014 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.394562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"
Sep 16 01:03:20.395014 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.394601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" event={"ID":"25211d3a-5e80-442a-8415-1d097007d5d5","Type":"ContainerDied","Data":"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"}
Sep 16 01:03:20.395014 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.394660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh" event={"ID":"25211d3a-5e80-442a-8415-1d097007d5d5","Type":"ContainerDied","Data":"be864b895dd3bd77fb3a33eb3e01cf2a8fb25a7da7c660b77a3df26e228043e1"}
Sep 16 01:03:20.395014 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.394684    2577 scope.go:117] "RemoveContainer" containerID="341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"
Sep 16 01:03:20.404936 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.404915    2577 scope.go:117] "RemoveContainer" containerID="341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"
Sep 16 01:03:20.405260 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:03:20.405234    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183\": container with ID starting with 341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183 not found: ID does not exist" containerID="341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"
Sep 16 01:03:20.405368 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.405270    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183"} err="failed to get container status \"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183\": rpc error: code = NotFound desc = could not find container \"341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183\": container with ID starting with 341fa4a2ef4448443cfc3f3d5ee11e0cc5f708bf10e4f0c916808133a57a3183 not found: ID does not exist"
Sep 16 01:03:20.411363 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.411339    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:03:20.414678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:20.414653    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c31c5-6b6455f547-w5zqh"]
Sep 16 01:03:22.334373 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:22.334341    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25211d3a-5e80-442a-8415-1d097007d5d5" path="/var/lib/kubelet/pods/25211d3a-5e80-442a-8415-1d097007d5d5/volumes"
Sep 16 01:03:24.991567 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:24.991536    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:03:24.992090 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:24.991822    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" podUID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" containerName="maas-api" containerID="cri-o://b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f" gracePeriod=30
Sep 16 01:03:25.251244 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.251175    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:03:25.371900 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.371864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") pod \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") "
Sep 16 01:03:25.372107 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.371935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca\") pod \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") "
Sep 16 01:03:25.372107 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.372031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs\") pod \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") "
Sep 16 01:03:25.372226 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.372153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np2nq\" (UniqueName: \"kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq\") pod \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\" (UID: \"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798\") "
Sep 16 01:03:25.372308 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.372275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" (UID: "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:03:25.372587 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.372551    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:25.375102 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.375063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" (UID: "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:25.375102 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.375097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" (UID: "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:25.375239 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.375176    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq" (OuterVolumeSpecName: "kube-api-access-np2nq") pod "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" (UID: "ab1b31b4-4f0b-4234-b1c3-2d6acd60a798"). InnerVolumeSpecName "kube-api-access-np2nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:25.433049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.431378    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" containerID="b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f" exitCode=0
Sep 16 01:03:25.433049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.431446    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"
Sep 16 01:03:25.433049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.431467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" event={"ID":"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798","Type":"ContainerDied","Data":"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"}
Sep 16 01:03:25.433049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.431553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz" event={"ID":"ab1b31b4-4f0b-4234-b1c3-2d6acd60a798","Type":"ContainerDied","Data":"6b942d851185e848fc59d9d0be191f0c5014dd44f3a180f801e767264032476d"}
Sep 16 01:03:25.433049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.431585    2577 scope.go:117] "RemoveContainer" containerID="b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"
Sep 16 01:03:25.448612 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.448591    2577 scope.go:117] "RemoveContainer" containerID="b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"
Sep 16 01:03:25.451631 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:03:25.451543    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f\": container with ID starting with b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f not found: ID does not exist" containerID="b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"
Sep 16 01:03:25.451631 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.451585    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f"} err="failed to get container status \"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f\": rpc error: code = NotFound desc = could not find container \"b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f\": container with ID starting with b20de92102e419dc1f2b85750ad179d0ac48a43291bcffb5534d4a442750f82f not found: ID does not exist"
Sep 16 01:03:25.473705 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.473677    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np2nq\" (UniqueName: \"kubernetes.io/projected/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-kube-api-access-np2nq\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:25.473705 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.473704    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:25.473705 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.473714    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:25.495682 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.495656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:03:25.508354 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:25.508325    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-eba3d3-5dfd66996-ghfdz"]
Sep 16 01:03:26.335429 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:26.335392    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" path="/var/lib/kubelet/pods/ab1b31b4-4f0b-4234-b1c3-2d6acd60a798/volumes"
Sep 16 01:03:39.327966 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:39.327925    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:03:44.579577 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.579481    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:03:44.580056 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.579775    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" podUID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" containerName="maas-api" containerID="cri-o://076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30" gracePeriod=30
Sep 16 01:03:44.853010 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.852979    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:03:44.995111 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca\") pod \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") "
Sep 16 01:03:44.995287 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs\") pod \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") "
Sep 16 01:03:44.995287 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln2h8\" (UniqueName: \"kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8\") pod \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") "
Sep 16 01:03:44.995368 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995293    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") pod \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\" (UID: \"1e482e96-3bfc-4ef3-8647-ae6dd3522634\") "
Sep 16 01:03:44.995523 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e482e96-3bfc-4ef3-8647-ae6dd3522634" (UID: "1e482e96-3bfc-4ef3-8647-ae6dd3522634"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:03:44.995653 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.995638    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e482e96-3bfc-4ef3-8647-ae6dd3522634-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:44.997571 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.997545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e482e96-3bfc-4ef3-8647-ae6dd3522634" (UID: "1e482e96-3bfc-4ef3-8647-ae6dd3522634"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:44.997678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.997662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e482e96-3bfc-4ef3-8647-ae6dd3522634" (UID: "1e482e96-3bfc-4ef3-8647-ae6dd3522634"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:44.997768 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:44.997740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8" (OuterVolumeSpecName: "kube-api-access-ln2h8") pod "1e482e96-3bfc-4ef3-8647-ae6dd3522634" (UID: "1e482e96-3bfc-4ef3-8647-ae6dd3522634"). InnerVolumeSpecName "kube-api-access-ln2h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:45.096692 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.096602    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:45.096692 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.096632    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2h8\" (UniqueName: \"kubernetes.io/projected/1e482e96-3bfc-4ef3-8647-ae6dd3522634-kube-api-access-ln2h8\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:45.096692 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.096641    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e482e96-3bfc-4ef3-8647-ae6dd3522634-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:45.553897 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.553860    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" containerID="076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30" exitCode=0
Sep 16 01:03:45.554101 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.553904    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"
Sep 16 01:03:45.554101 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.553917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" event={"ID":"1e482e96-3bfc-4ef3-8647-ae6dd3522634","Type":"ContainerDied","Data":"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"}
Sep 16 01:03:45.554101 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.553979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n" event={"ID":"1e482e96-3bfc-4ef3-8647-ae6dd3522634","Type":"ContainerDied","Data":"25a8e6c20aeaaf70c13fe587340d687f0272583e7ba0a5cf2bc218f7991f4086"}
Sep 16 01:03:45.554101 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.554014    2577 scope.go:117] "RemoveContainer" containerID="076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"
Sep 16 01:03:45.565151 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.565128    2577 scope.go:117] "RemoveContainer" containerID="076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"
Sep 16 01:03:45.565504 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:03:45.565473    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30\": container with ID starting with 076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30 not found: ID does not exist" containerID="076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"
Sep 16 01:03:45.565620 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.565518    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30"} err="failed to get container status \"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30\": rpc error: code = NotFound desc = could not find container \"076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30\": container with ID starting with 076858d11ae28ae3c2ef53ca5a286559afadec6e4fa20a0b126a7064971fad30 not found: ID does not exist"
Sep 16 01:03:45.576426 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.576401    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:03:45.580243 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:45.580217    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-761551-6c8c56856d-6rc5n"]
Sep 16 01:03:46.334676 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:46.334635    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" path="/var/lib/kubelet/pods/1e482e96-3bfc-4ef3-8647-ae6dd3522634/volumes"
Sep 16 01:03:50.458669 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:50.458640    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:03:50.469298 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:50.469276    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:03:52.544618 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.544579    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:03:52.545559 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.545505    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" podUID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" containerName="maas-api" containerID="cri-o://e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1" gracePeriod=30
Sep 16 01:03:52.838483 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.838450    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:03:52.995397 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.995362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs\") pod \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") "
Sep 16 01:03:52.995590 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.995426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") pod \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") "
Sep 16 01:03:52.995590 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.995493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca\") pod \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") "
Sep 16 01:03:52.995734 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.995707    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4fjg\" (UniqueName: \"kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg\") pod \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\" (UID: \"9c92a798-f9be-4f5a-b309-5e10ff9cebf9\") "
Sep 16 01:03:52.995887 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.995858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c92a798-f9be-4f5a-b309-5e10ff9cebf9" (UID: "9c92a798-f9be-4f5a-b309-5e10ff9cebf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:03:52.996165 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.996145    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:52.997794 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.997768    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c92a798-f9be-4f5a-b309-5e10ff9cebf9" (UID: "9c92a798-f9be-4f5a-b309-5e10ff9cebf9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:52.998414 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.998386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c92a798-f9be-4f5a-b309-5e10ff9cebf9" (UID: "9c92a798-f9be-4f5a-b309-5e10ff9cebf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:03:52.998414 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:52.998401    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg" (OuterVolumeSpecName: "kube-api-access-x4fjg") pod "9c92a798-f9be-4f5a-b309-5e10ff9cebf9" (UID: "9c92a798-f9be-4f5a-b309-5e10ff9cebf9"). InnerVolumeSpecName "kube-api-access-x4fjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:03:53.097288 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.097186    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:53.097288 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.097223    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:53.097288 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.097235    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4fjg\" (UniqueName: \"kubernetes.io/projected/9c92a798-f9be-4f5a-b309-5e10ff9cebf9-kube-api-access-x4fjg\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:03:53.612460 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.612422    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" containerID="e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1" exitCode=0
Sep 16 01:03:53.612973 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.612469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"
Sep 16 01:03:53.612973 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.612505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" event={"ID":"9c92a798-f9be-4f5a-b309-5e10ff9cebf9","Type":"ContainerDied","Data":"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"}
Sep 16 01:03:53.612973 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.612573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk" event={"ID":"9c92a798-f9be-4f5a-b309-5e10ff9cebf9","Type":"ContainerDied","Data":"192dc659cf5d8aca7a172ff65ea7ecda13ada0c1010918e54409fa67eb99b109"}
Sep 16 01:03:53.612973 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.612609    2577 scope.go:117] "RemoveContainer" containerID="e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"
Sep 16 01:03:53.623306 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.623287    2577 scope.go:117] "RemoveContainer" containerID="e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"
Sep 16 01:03:53.623609 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:03:53.623589    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1\": container with ID starting with e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1 not found: ID does not exist" containerID="e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"
Sep 16 01:03:53.623673 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.623623    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1"} err="failed to get container status \"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1\": rpc error: code = NotFound desc = could not find container \"e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1\": container with ID starting with e398ee3d502ed0c66bf7713bae4938029a3c205c2946eec2be395e088b4e64e1 not found: ID does not exist"
Sep 16 01:03:53.635138 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.635112    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:03:53.639970 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:53.639930    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-715834-6b699b49bc-s5zsk"]
Sep 16 01:03:54.336685 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:03:54.336646    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" path="/var/lib/kubelet/pods/9c92a798-f9be-4f5a-b309-5e10ff9cebf9/volumes"
Sep 16 01:04:02.242451 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.242415    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 01:04:02.242923 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.242693    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" podUID="31c74a99-8124-47e9-a5e0-71e17f342925" containerName="maas-api" containerID="cri-o://db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7" gracePeriod=30
Sep 16 01:04:02.327726 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.327690    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:04:02.508027 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.507998    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 01:04:02.596809 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.596772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs\") pod \"31c74a99-8124-47e9-a5e0-71e17f342925\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") "
Sep 16 01:04:02.596809 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.596811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls\") pod \"31c74a99-8124-47e9-a5e0-71e17f342925\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") "
Sep 16 01:04:02.597077 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.596833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpj79\" (UniqueName: \"kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79\") pod \"31c74a99-8124-47e9-a5e0-71e17f342925\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") "
Sep 16 01:04:02.597077 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.596946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca\") pod \"31c74a99-8124-47e9-a5e0-71e17f342925\" (UID: \"31c74a99-8124-47e9-a5e0-71e17f342925\") "
Sep 16 01:04:02.597349 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.597321    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31c74a99-8124-47e9-a5e0-71e17f342925" (UID: "31c74a99-8124-47e9-a5e0-71e17f342925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:04:02.599516 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.599489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31c74a99-8124-47e9-a5e0-71e17f342925" (UID: "31c74a99-8124-47e9-a5e0-71e17f342925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:02.599516 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.599499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79" (OuterVolumeSpecName: "kube-api-access-fpj79") pod "31c74a99-8124-47e9-a5e0-71e17f342925" (UID: "31c74a99-8124-47e9-a5e0-71e17f342925"). InnerVolumeSpecName "kube-api-access-fpj79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:02.599665 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.599529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31c74a99-8124-47e9-a5e0-71e17f342925" (UID: "31c74a99-8124-47e9-a5e0-71e17f342925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:02.674015 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.673982    2577 generic.go:356] "Generic (PLEG): container finished" podID="31c74a99-8124-47e9-a5e0-71e17f342925" containerID="db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7" exitCode=0
Sep 16 01:04:02.674190 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.674040    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"
Sep 16 01:04:02.674190 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.674055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" event={"ID":"31c74a99-8124-47e9-a5e0-71e17f342925","Type":"ContainerDied","Data":"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"}
Sep 16 01:04:02.674190 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.674099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw" event={"ID":"31c74a99-8124-47e9-a5e0-71e17f342925","Type":"ContainerDied","Data":"febdc6fa9f67c31c378953588540187b94b5c0700c4d19ee635b89ce1824d7a2"}
Sep 16 01:04:02.674190 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.674121    2577 scope.go:117] "RemoveContainer" containerID="db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"
Sep 16 01:04:02.684719 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.684542    2577 scope.go:117] "RemoveContainer" containerID="db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"
Sep 16 01:04:02.685000 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:02.684974    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7\": container with ID starting with db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7 not found: ID does not exist" containerID="db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"
Sep 16 01:04:02.685083 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.685012    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7"} err="failed to get container status \"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7\": rpc error: code = NotFound desc = could not find container \"db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7\": container with ID starting with db0d734cb670a4baacd3ed27017fc2f07879e6b764458222b5eae7479ed738a7 not found: ID does not exist"
Sep 16 01:04:02.698213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.698184    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:02.698213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.698209    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31c74a99-8124-47e9-a5e0-71e17f342925-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:02.698213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.698218    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpj79\" (UniqueName: \"kubernetes.io/projected/31c74a99-8124-47e9-a5e0-71e17f342925-kube-api-access-fpj79\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:02.698213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.698228    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c74a99-8124-47e9-a5e0-71e17f342925-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:02.698484 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.698286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 01:04:02.703760 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:02.703730    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3462ea-5798c8697c-zskzw"]
Sep 16 01:04:04.335346 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:04.335299    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c74a99-8124-47e9-a5e0-71e17f342925" path="/var/lib/kubelet/pods/31c74a99-8124-47e9-a5e0-71e17f342925/volumes"
Sep 16 01:04:08.795675 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:08.795637    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 01:04:08.796109 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:08.795872    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" containerID="cri-o://b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985" gracePeriod=30
Sep 16 01:04:11.131118 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:11.131079    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 01:04:11.131637 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:11.131145    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 01:04:14.064056 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.063989    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 01:04:14.219143 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219143 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219358    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbjjk\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219399 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token\") pod \"b1096bd0-3b30-4565-84d7-833f9b0f405d\" (UID: \"b1096bd0-3b30-4565-84d7-833f9b0f405d\") "
Sep 16 01:04:14.219758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:14.219758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:14.219758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:14.219931 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219810    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.219931 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219830    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.219931 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:14.219931 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219845    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.220361 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.219973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:04:14.221820 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.221794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk" (OuterVolumeSpecName: "kube-api-access-pbjjk") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "kube-api-access-pbjjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:14.221940 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.221871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:04:14.221940 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.221893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:14.221940 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.221902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1096bd0-3b30-4565-84d7-833f9b0f405d" (UID: "b1096bd0-3b30-4565-84d7-833f9b0f405d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321144    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1096bd0-3b30-4565-84d7-833f9b0f405d-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321176    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321186    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321195    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjjk\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-kube-api-access-pbjjk\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321204    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.321237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.321212    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1096bd0-3b30-4565-84d7-833f9b0f405d-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:14.743622 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.743530    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerID="b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985" exitCode=0
Sep 16 01:04:14.743622 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.743578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"
Sep 16 01:04:14.743874 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.743614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerDied","Data":"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"}
Sep 16 01:04:14.743874 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.743687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8" event={"ID":"b1096bd0-3b30-4565-84d7-833f9b0f405d","Type":"ContainerDied","Data":"4181bb612811eced7bf6b67c28786e2b5ab8f30d87e18db64bea93e559a160ae"}
Sep 16 01:04:14.743874 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.743717    2577 scope.go:117] "RemoveContainer" containerID="b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"
Sep 16 01:04:14.753344 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.753323    2577 scope.go:117] "RemoveContainer" containerID="3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"
Sep 16 01:04:14.765482 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.765462    2577 scope.go:117] "RemoveContainer" containerID="b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"
Sep 16 01:04:14.765799 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.765774    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 01:04:14.765799 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:14.765788    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985\": container with ID starting with b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985 not found: ID does not exist" containerID="b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"
Sep 16 01:04:14.766021 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.765815    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985"} err="failed to get container status \"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985\": rpc error: code = NotFound desc = could not find container \"b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985\": container with ID starting with b51a56e4810640ba61cf19c1c84ae3a45b357bcc240998fee5b0e22ad6fae985 not found: ID does not exist"
Sep 16 01:04:14.766021 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.765836    2577 scope.go:117] "RemoveContainer" containerID="3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"
Sep 16 01:04:14.766142 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:14.766124    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf\": container with ID starting with 3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf not found: ID does not exist" containerID="3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"
Sep 16 01:04:14.766201 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.766143    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf"} err="failed to get container status \"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf\": rpc error: code = NotFound desc = could not find container \"3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf\": container with ID starting with 3fc9348e4a3a227a207ca599bcd384ff4d26cee41cf8ee627b4e1c3a8eb91baf not found: ID does not exist"
Sep 16 01:04:14.768234 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:14.768212    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3462ea-openshift-default-dd886877d-cs2w8"]
Sep 16 01:04:16.334873 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:16.334835    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" path="/var/lib/kubelet/pods/b1096bd0-3b30-4565-84d7-833f9b0f405d/volumes"
Sep 16 01:04:19.627837 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.627787    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 01:04:19.628382 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.628094    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" podUID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" containerName="maas-api" containerID="cri-o://e2bbcad06b3b0d9c13b0a3e633b84da89cfd009d5ca902a65fd8b81a92f4a1d2" gracePeriod=30
Sep 16 01:04:19.777697 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.777659    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" containerID="e2bbcad06b3b0d9c13b0a3e633b84da89cfd009d5ca902a65fd8b81a92f4a1d2" exitCode=0
Sep 16 01:04:19.777852 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.777711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" event={"ID":"ef4e0c6e-e721-4da3-b706-3493b2174e2b","Type":"ContainerDied","Data":"e2bbcad06b3b0d9c13b0a3e633b84da89cfd009d5ca902a65fd8b81a92f4a1d2"}
Sep 16 01:04:19.898722 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.898687    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 01:04:19.980315 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.980276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs\") pod \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") "
Sep 16 01:04:19.980497 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.980419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") pod \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") "
Sep 16 01:04:19.980497 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.980476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca\") pod \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") "
Sep 16 01:04:19.980579 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.980527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnvv4\" (UniqueName: \"kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4\") pod \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\" (UID: \"ef4e0c6e-e721-4da3-b706-3493b2174e2b\") "
Sep 16 01:04:19.980896 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.980866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef4e0c6e-e721-4da3-b706-3493b2174e2b" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:04:19.982828 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.982793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef4e0c6e-e721-4da3-b706-3493b2174e2b" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:19.982828 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.982805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4" (OuterVolumeSpecName: "kube-api-access-mnvv4") pod "ef4e0c6e-e721-4da3-b706-3493b2174e2b" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b"). InnerVolumeSpecName "kube-api-access-mnvv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:19.982995 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:19.982836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef4e0c6e-e721-4da3-b706-3493b2174e2b" (UID: "ef4e0c6e-e721-4da3-b706-3493b2174e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:20.081758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.081720    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:20.081758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.081753    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4e0c6e-e721-4da3-b706-3493b2174e2b-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:20.081758 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.081765    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvv4\" (UniqueName: \"kubernetes.io/projected/ef4e0c6e-e721-4da3-b706-3493b2174e2b-kube-api-access-mnvv4\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:20.082067 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.081774    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4e0c6e-e721-4da3-b706-3493b2174e2b-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:20.785315 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.785281    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"
Sep 16 01:04:20.785820 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.785321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628" event={"ID":"ef4e0c6e-e721-4da3-b706-3493b2174e2b","Type":"ContainerDied","Data":"524b62a11e31f1ae4c1b8d4a5ec3a581c9df7805e57c352fed1f8eecbcfa6e52"}
Sep 16 01:04:20.785820 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.785388    2577 scope.go:117] "RemoveContainer" containerID="e2bbcad06b3b0d9c13b0a3e633b84da89cfd009d5ca902a65fd8b81a92f4a1d2"
Sep 16 01:04:20.803745 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.803715    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 01:04:20.807510 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:20.807483    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-409cde-6679db7f44-s6628"]
Sep 16 01:04:22.334060 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:22.334025    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" path="/var/lib/kubelet/pods/ef4e0c6e-e721-4da3-b706-3493b2174e2b/volumes"
Sep 16 01:04:41.132391 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:41.132351    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:04:41.132916 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:41.132566    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy" containerID="cri-o://ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87" gracePeriod=30
Sep 16 01:04:46.396589 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.396519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:04:46.549687 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.549878 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.549878 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.549878 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.549878 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.549878 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv5jg\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.550237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.549942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.550237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.550237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs\") pod \"52cf260b-293a-4718-88f2-f8790dc97434\" (UID: \"52cf260b-293a-4718-88f2-f8790dc97434\") "
Sep 16 01:04:46.550237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:46.550237 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:46.550504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:46.550504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:04:46.550504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550458    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.550504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550477    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.550504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550491    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.550704 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.550601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data" (OuterVolumeSpecName: "istio-data") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:46.552226 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.552200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:04:46.552405 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.552383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:04:46.552720 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.552701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token" (OuterVolumeSpecName: "istio-token") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:46.552764 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.552731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg" (OuterVolumeSpecName: "kube-api-access-wv5jg") pod "52cf260b-293a-4718-88f2-f8790dc97434" (UID: "52cf260b-293a-4718-88f2-f8790dc97434"). InnerVolumeSpecName "kube-api-access-wv5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651131    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cf260b-293a-4718-88f2-f8790dc97434-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651159    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651168    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cf260b-293a-4718-88f2-f8790dc97434-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651178    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv5jg\" (UniqueName: \"kubernetes.io/projected/52cf260b-293a-4718-88f2-f8790dc97434-kube-api-access-wv5jg\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651187    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.651215 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.651196    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cf260b-293a-4718-88f2-f8790dc97434-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:46.933893 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.933798    2577 generic.go:356] "Generic (PLEG): container finished" podID="52cf260b-293a-4718-88f2-f8790dc97434" containerID="ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87" exitCode=0
Sep 16 01:04:46.933893 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.933857    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"
Sep 16 01:04:46.933893 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.933881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerDied","Data":"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"}
Sep 16 01:04:46.934207 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.933944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd" event={"ID":"52cf260b-293a-4718-88f2-f8790dc97434","Type":"ContainerDied","Data":"851b024de40f5626b043f0891089c45c5ecc9db11b5dc5f1ba4494f16321bc59"}
Sep 16 01:04:46.934207 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.933985    2577 scope.go:117] "RemoveContainer" containerID="ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"
Sep 16 01:04:46.945271 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.945254    2577 scope.go:117] "RemoveContainer" containerID="0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"
Sep 16 01:04:46.957458 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.957431    2577 scope.go:117] "RemoveContainer" containerID="ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"
Sep 16 01:04:46.957796 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:46.957775    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87\": container with ID starting with ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87 not found: ID does not exist" containerID="ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"
Sep 16 01:04:46.957888 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.957804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87"} err="failed to get container status \"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87\": rpc error: code = NotFound desc = could not find container \"ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87\": container with ID starting with ee3e056e6a5367c59acd4428cef93acb228c165f847a861658334fcb56cfba87 not found: ID does not exist"
Sep 16 01:04:46.957888 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.957827    2577 scope.go:117] "RemoveContainer" containerID="0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"
Sep 16 01:04:46.958151 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:46.958132    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2\": container with ID starting with 0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2 not found: ID does not exist" containerID="0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"
Sep 16 01:04:46.958197 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.958159    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2"} err="failed to get container status \"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2\": rpc error: code = NotFound desc = could not find container \"0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2\": container with ID starting with 0cb498805dc9808faf623b6f4ad5f0e1b71ee77ce22d79955b90a77701df9eb2 not found: ID does not exist"
Sep 16 01:04:46.958902 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.958883    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:04:46.962763 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:46.962740    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-715834-openshift-default-658f57b8b6-wd9hd"]
Sep 16 01:04:48.333868 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:48.333830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52cf260b-293a-4718-88f2-f8790dc97434" path="/var/lib/kubelet/pods/52cf260b-293a-4718-88f2-f8790dc97434/volumes"
Sep 16 01:04:51.940096 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:51.940057    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:04:51.940900 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:51.940865    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" podUID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" containerName="maas-api" containerID="cri-o://a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382" gracePeriod=30
Sep 16 01:04:52.196395 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.196318    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:04:52.304730 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.304698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs\") pod \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") "
Sep 16 01:04:52.304909 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.304751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca\") pod \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") "
Sep 16 01:04:52.304909 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.304791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf6s8\" (UniqueName: \"kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8\") pod \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") "
Sep 16 01:04:52.304909 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.304830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") pod \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\" (UID: \"f18a5872-a3ec-43ee-be18-f12541ecfcbf\") "
Sep 16 01:04:52.305272 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.305231    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f18a5872-a3ec-43ee-be18-f12541ecfcbf" (UID: "f18a5872-a3ec-43ee-be18-f12541ecfcbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:04:52.307164 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.307140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f18a5872-a3ec-43ee-be18-f12541ecfcbf" (UID: "f18a5872-a3ec-43ee-be18-f12541ecfcbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:52.307313 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.307177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8" (OuterVolumeSpecName: "kube-api-access-qf6s8") pod "f18a5872-a3ec-43ee-be18-f12541ecfcbf" (UID: "f18a5872-a3ec-43ee-be18-f12541ecfcbf"). InnerVolumeSpecName "kube-api-access-qf6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:04:52.307313 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.307204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f18a5872-a3ec-43ee-be18-f12541ecfcbf" (UID: "f18a5872-a3ec-43ee-be18-f12541ecfcbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:04:52.405636 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.405605    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:52.405636 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.405634    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a5872-a3ec-43ee-be18-f12541ecfcbf-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:52.405636 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.405644    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6s8\" (UniqueName: \"kubernetes.io/projected/f18a5872-a3ec-43ee-be18-f12541ecfcbf-kube-api-access-qf6s8\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:52.405866 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.405653    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a5872-a3ec-43ee-be18-f12541ecfcbf-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:04:52.972273 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.972235    2577 generic.go:356] "Generic (PLEG): container finished" podID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" containerID="a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382" exitCode=0
Sep 16 01:04:52.972779 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.972310    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"
Sep 16 01:04:52.972779 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.972314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" event={"ID":"f18a5872-a3ec-43ee-be18-f12541ecfcbf","Type":"ContainerDied","Data":"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"}
Sep 16 01:04:52.972779 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.972407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4" event={"ID":"f18a5872-a3ec-43ee-be18-f12541ecfcbf","Type":"ContainerDied","Data":"00cc6f87dc6c640294051154a80cd1322f54cadf1b5dfdbbebf21d92ed73f933"}
Sep 16 01:04:52.972779 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.972433    2577 scope.go:117] "RemoveContainer" containerID="a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"
Sep 16 01:04:52.982071 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.982042    2577 scope.go:117] "RemoveContainer" containerID="a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"
Sep 16 01:04:52.982324 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:04:52.982306    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382\": container with ID starting with a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382 not found: ID does not exist" containerID="a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"
Sep 16 01:04:52.982380 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.982334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382"} err="failed to get container status \"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382\": rpc error: code = NotFound desc = could not find container \"a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382\": container with ID starting with a3ccf28b3c983032afc24815166880a8261252d6b3d8fbf4f20b6b471a211382 not found: ID does not exist"
Sep 16 01:04:52.991050 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.991025    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:04:52.995158 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:52.995136    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e8a7b-6d8578f675-cwwz4"]
Sep 16 01:04:54.333612 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:04:54.333578    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" path="/var/lib/kubelet/pods/f18a5872-a3ec-43ee-be18-f12541ecfcbf/volumes"
Sep 16 01:05:09.328143 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.328105    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:05:09.512668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.512632    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 01:05:09.512934 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.512906    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" podUID="6c258f94-0cb9-4a70-8981-d1079ccc135b" containerName="maas-api" containerID="cri-o://dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df" gracePeriod=30
Sep 16 01:05:09.775108 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.775079    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 01:05:09.873503 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.873450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca\") pod \"6c258f94-0cb9-4a70-8981-d1079ccc135b\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") "
Sep 16 01:05:09.873686 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.873528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2scvc\" (UniqueName: \"kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc\") pod \"6c258f94-0cb9-4a70-8981-d1079ccc135b\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") "
Sep 16 01:05:09.873686 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.873590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs\") pod \"6c258f94-0cb9-4a70-8981-d1079ccc135b\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") "
Sep 16 01:05:09.873686 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.873614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") pod \"6c258f94-0cb9-4a70-8981-d1079ccc135b\" (UID: \"6c258f94-0cb9-4a70-8981-d1079ccc135b\") "
Sep 16 01:05:09.873926 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.873873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c258f94-0cb9-4a70-8981-d1079ccc135b" (UID: "6c258f94-0cb9-4a70-8981-d1079ccc135b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:09.874177 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.874051    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c258f94-0cb9-4a70-8981-d1079ccc135b-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.876539 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.876479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c258f94-0cb9-4a70-8981-d1079ccc135b" (UID: "6c258f94-0cb9-4a70-8981-d1079ccc135b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:09.876813 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.876785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc" (OuterVolumeSpecName: "kube-api-access-2scvc") pod "6c258f94-0cb9-4a70-8981-d1079ccc135b" (UID: "6c258f94-0cb9-4a70-8981-d1079ccc135b"). InnerVolumeSpecName "kube-api-access-2scvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:09.876941 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.876835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c258f94-0cb9-4a70-8981-d1079ccc135b" (UID: "6c258f94-0cb9-4a70-8981-d1079ccc135b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:09.975257 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.975224    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.975257 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.975254    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c258f94-0cb9-4a70-8981-d1079ccc135b-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:09.975504 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:09.975265    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2scvc\" (UniqueName: \"kubernetes.io/projected/6c258f94-0cb9-4a70-8981-d1079ccc135b-kube-api-access-2scvc\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:10.072514 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.072479    2577 generic.go:356] "Generic (PLEG): container finished" podID="6c258f94-0cb9-4a70-8981-d1079ccc135b" containerID="dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df" exitCode=0
Sep 16 01:05:10.072678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.072525    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"
Sep 16 01:05:10.072678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.072562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" event={"ID":"6c258f94-0cb9-4a70-8981-d1079ccc135b","Type":"ContainerDied","Data":"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"}
Sep 16 01:05:10.072678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.072615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t" event={"ID":"6c258f94-0cb9-4a70-8981-d1079ccc135b","Type":"ContainerDied","Data":"0ea258297228d7ff06dab25ebdc9a7253c93ab42b2bf663e22f4a585321edaea"}
Sep 16 01:05:10.072678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.072639    2577 scope.go:117] "RemoveContainer" containerID="dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"
Sep 16 01:05:10.082456 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.082439    2577 scope.go:117] "RemoveContainer" containerID="dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"
Sep 16 01:05:10.082753 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:05:10.082737    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df\": container with ID starting with dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df not found: ID does not exist" containerID="dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"
Sep 16 01:05:10.082804 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.082761    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df"} err="failed to get container status \"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df\": rpc error: code = NotFound desc = could not find container \"dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df\": container with ID starting with dc6c0596b912b1611a0540d3b66cd67cfcf2bd6aad70dce56331cf967e6224df not found: ID does not exist"
Sep 16 01:05:10.095046 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.094936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 01:05:10.098537 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.098513    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-108960-589444b599-tcf4t"]
Sep 16 01:05:10.335022 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:10.334985    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c258f94-0cb9-4a70-8981-d1079ccc135b" path="/var/lib/kubelet/pods/6c258f94-0cb9-4a70-8981-d1079ccc135b/volumes"
Sep 16 01:05:16.156283 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:16.156198    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 01:05:16.156735 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:16.156427    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy" containerID="cri-o://624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534" gracePeriod=30
Sep 16 01:05:21.327799 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.327762    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:05:21.409041 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.409013    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 01:05:21.480126 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480126 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480126 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480350 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480410 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:21.480410 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480390    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480498 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480498 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480498 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mndsm\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm\") pod \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\" (UID: \"e688e66c-2e95-4b5d-8448-f1e0888efaa7\") "
Sep 16 01:05:21.480668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:21.480777 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:21.480777 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:21.480922 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:21.480922 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480915    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-credential-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.481081 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480933    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-socket\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.481081 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480943    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-workload-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.481081 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.480978    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-data\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.482850 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.482816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 01:05:21.482850 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.482820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm" (OuterVolumeSpecName: "kube-api-access-mndsm") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "kube-api-access-mndsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:21.483081 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.483062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 01:05:21.483167 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.483146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e688e66c-2e95-4b5d-8448-f1e0888efaa7" (UID: "e688e66c-2e95-4b5d-8448-f1e0888efaa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:21.582088 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.582047    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istiod-ca-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.582088 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.582079    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndsm\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-kube-api-access-mndsm\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.582088 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.582090    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-podinfo\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.582088 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.582099    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-envoy\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:21.582378 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:21.582108    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e688e66c-2e95-4b5d-8448-f1e0888efaa7-istio-token\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:22.139712 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.139678    2577 generic.go:356] "Generic (PLEG): container finished" podID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerID="624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534" exitCode=0
Sep 16 01:05:22.139901 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.139784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"
Sep 16 01:05:22.139901 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.139797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerDied","Data":"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"}
Sep 16 01:05:22.139901 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.139842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr" event={"ID":"e688e66c-2e95-4b5d-8448-f1e0888efaa7","Type":"ContainerDied","Data":"03888b5227815826d1e4cded05361ae18574b54a869574ce81e5b5fb4ff6c31f"}
Sep 16 01:05:22.139901 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.139863    2577 scope.go:117] "RemoveContainer" containerID="624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"
Sep 16 01:05:22.153128 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.153106    2577 scope.go:117] "RemoveContainer" containerID="62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"
Sep 16 01:05:22.162415 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.162401    2577 scope.go:117] "RemoveContainer" containerID="624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"
Sep 16 01:05:22.162630 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.162607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 01:05:22.162714 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:05:22.162695    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534\": container with ID starting with 624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534 not found: ID does not exist" containerID="624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"
Sep 16 01:05:22.162755 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.162723    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534"} err="failed to get container status \"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534\": rpc error: code = NotFound desc = could not find container \"624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534\": container with ID starting with 624cb18f9299c6ec2ae86f1cc594596415baad21b6d24d2040602e5634582534 not found: ID does not exist"
Sep 16 01:05:22.162755 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.162741    2577 scope.go:117] "RemoveContainer" containerID="62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"
Sep 16 01:05:22.163026 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:05:22.163008    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc\": container with ID starting with 62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc not found: ID does not exist" containerID="62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"
Sep 16 01:05:22.163091 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.163035    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc"} err="failed to get container status \"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc\": rpc error: code = NotFound desc = could not find container \"62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc\": container with ID starting with 62f8aa723b8d8a93571f077a7fb4639772657802696cfe69d572667d09fcaebc not found: ID does not exist"
Sep 16 01:05:22.169072 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.169048    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-108960-openshift-default-6c7576fb74-k66jr"]
Sep 16 01:05:22.335020 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:22.334978    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" path="/var/lib/kubelet/pods/e688e66c-2e95-4b5d-8448-f1e0888efaa7/volumes"
Sep 16 01:05:26.955613 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:26.955572    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 01:05:26.956150 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:26.955873    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" podUID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" containerName="maas-api" containerID="cri-o://00bde1b444424dee2f7a5c53c730082c37974594fb43523c1996909c4d5e815f" gracePeriod=30
Sep 16 01:05:27.170260 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.170224    2577 generic.go:356] "Generic (PLEG): container finished" podID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" containerID="00bde1b444424dee2f7a5c53c730082c37974594fb43523c1996909c4d5e815f" exitCode=0
Sep 16 01:05:27.170465 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.170311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" event={"ID":"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562","Type":"ContainerDied","Data":"00bde1b444424dee2f7a5c53c730082c37974594fb43523c1996909c4d5e815f"}
Sep 16 01:05:27.221999 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.221966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 01:05:27.336084 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.336041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p9mc\" (UniqueName: \"kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc\") pod \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") "
Sep 16 01:05:27.336301 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.336183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") pod \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") "
Sep 16 01:05:27.336301 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.336207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs\") pod \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") "
Sep 16 01:05:27.336301 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.336241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca\") pod \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\" (UID: \"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562\") "
Sep 16 01:05:27.336739 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.336679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" (UID: "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 01:05:27.338447 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.338419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" (UID: "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:27.338978 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.338928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" (UID: "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:27.339088 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.338935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc" (OuterVolumeSpecName: "kube-api-access-5p9mc") pod "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" (UID: "b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562"). InnerVolumeSpecName "kube-api-access-5p9mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:27.437796 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.437762    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-maas-api-tls\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:27.437796 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.437791    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:27.437796 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.437800    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-openshift-service-ca\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:27.438074 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:27.437810    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9mc\" (UniqueName: \"kubernetes.io/projected/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562-kube-api-access-5p9mc\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:28.177869 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.177835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"
Sep 16 01:05:28.178359 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.177862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw" event={"ID":"b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562","Type":"ContainerDied","Data":"fcccc7073030e548931e2cac7dc17694a21ec8539ed6f0e783e0e612749c0d28"}
Sep 16 01:05:28.178359 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.177931    2577 scope.go:117] "RemoveContainer" containerID="00bde1b444424dee2f7a5c53c730082c37974594fb43523c1996909c4d5e815f"
Sep 16 01:05:28.209362 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.209333    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 01:05:28.224700 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.224676    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2fa86d-7b674d77b5-k9lrw"]
Sep 16 01:05:28.334331 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:28.334293    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" path="/var/lib/kubelet/pods/b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562/volumes"
Sep 16 01:05:36.831544 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:36.831503    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 01:05:36.832051 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:36.831727    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" podUID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" containerName="manager" containerID="cri-o://2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e" gracePeriod=30
Sep 16 01:05:37.087082 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.087023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 01:05:37.230439 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.230406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs\") pod \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") "
Sep 16 01:05:37.230610 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.230455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert\") pod \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") "
Sep 16 01:05:37.230610 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.230496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6gfk\" (UniqueName: \"kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk\") pod \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\" (UID: \"0108d3a9-9020-4e42-b146-0b9a1f9cb809\") "
Sep 16 01:05:37.233043 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0108d3a9-9020-4e42-b146-0b9a1f9cb809" (UID: "0108d3a9-9020-4e42-b146-0b9a1f9cb809"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:37.233160 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk" (OuterVolumeSpecName: "kube-api-access-d6gfk") pod "0108d3a9-9020-4e42-b146-0b9a1f9cb809" (UID: "0108d3a9-9020-4e42-b146-0b9a1f9cb809"). InnerVolumeSpecName "kube-api-access-d6gfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:05:37.233160 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0108d3a9-9020-4e42-b146-0b9a1f9cb809" (UID: "0108d3a9-9020-4e42-b146-0b9a1f9cb809"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:05:37.233440 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233417    2577 generic.go:356] "Generic (PLEG): container finished" podID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" containerID="2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e" exitCode=0
Sep 16 01:05:37.233496 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xb6cv"
Sep 16 01:05:37.233496 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" event={"ID":"0108d3a9-9020-4e42-b146-0b9a1f9cb809","Type":"ContainerDied","Data":"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"}
Sep 16 01:05:37.233572 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xb6cv" event={"ID":"0108d3a9-9020-4e42-b146-0b9a1f9cb809","Type":"ContainerDied","Data":"96de4f773b23a1598f4711ed8c7f8ee9859b79a9f081c226388e1b9f8627eb62"}
Sep 16 01:05:37.233572 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.233543    2577 scope.go:117] "RemoveContainer" containerID="2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"
Sep 16 01:05:37.252616 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.252584    2577 scope.go:117] "RemoveContainer" containerID="2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"
Sep 16 01:05:37.252969 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:05:37.252930    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e\": container with ID starting with 2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e not found: ID does not exist" containerID="2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"
Sep 16 01:05:37.253035 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.252975    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e"} err="failed to get container status \"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e\": rpc error: code = NotFound desc = could not find container \"2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e\": container with ID starting with 2fb69117989a95badf22dc29252472badac7e97252bc6d5b73b6c6e6e3d5326e not found: ID does not exist"
Sep 16 01:05:37.266018 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.265989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 01:05:37.271364 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.271338    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xb6cv"]
Sep 16 01:05:37.331830 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.331794    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:37.331830 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.331825    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0108d3a9-9020-4e42-b146-0b9a1f9cb809-webhook-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:37.332025 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:37.331858    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gfk\" (UniqueName: \"kubernetes.io/projected/0108d3a9-9020-4e42-b146-0b9a1f9cb809-kube-api-access-d6gfk\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:05:38.334485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:05:38.334451    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" path="/var/lib/kubelet/pods/0108d3a9-9020-4e42-b146-0b9a1f9cb809/volumes"
Sep 16 01:06:16.328220 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:16.328172    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:06:38.167092 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:38.167485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25211d3a-5e80-442a-8415-1d097007d5d5" containerName="maas-api"
Sep 16 01:06:38.167485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167458    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25211d3a-5e80-442a-8415-1d097007d5d5" containerName="maas-api"
Sep 16 01:06:38.167485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167472    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" containerName="maas-api"
Sep 16 01:06:38.167485 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167480    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167498    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167518    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167526    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167556    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167561    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167580    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167597    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" containerName="maas-api"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167607    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" containerName="manager"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" containerName="manager"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167633    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.167634 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167642    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167662    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c258f94-0cb9-4a70-8981-d1079ccc135b" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167670    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c258f94-0cb9-4a70-8981-d1079ccc135b" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167678    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167697    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c74a99-8124-47e9-a5e0-71e17f342925" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167706    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c74a99-8124-47e9-a5e0-71e17f342925" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167797    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c74a99-8124-47e9-a5e0-71e17f342925" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167815    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c92a798-f9be-4f5a-b309-5e10ff9cebf9" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167838    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0108d3a9-9020-4e42-b146-0b9a1f9cb809" containerName="manager"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167845    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167851    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fd31ea-5a1a-4cb3-ba4e-64186d3c5562" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167858    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167865    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c258f94-0cb9-4a70-8981-d1079ccc135b" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e482e96-3bfc-4ef3-8647-ae6dd3522634" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167879    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18a5872-a3ec-43ee-be18-f12541ecfcbf" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167887    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef4e0c6e-e721-4da3-b706-3493b2174e2b" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167896    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1b31b4-4f0b-4234-b1c3-2d6acd60a798" containerName="maas-api"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167906    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1096bd0-3b30-4565-84d7-833f9b0f405d" containerName="istio-proxy"
Sep 16 01:06:38.168337 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.167915    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25211d3a-5e80-442a-8415-1d097007d5d5" containerName="maas-api"
Sep 16 01:06:38.175533 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.173448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.178118 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.177768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 01:06:38.178118 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.177768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 01:06:38.178332 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.178202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 01:06:38.178401 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.178371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk8jt\""
Sep 16 01:06:38.181043 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.181007    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:38.277517 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.277486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.277668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.277542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pp4\" (UniqueName: \"kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.277668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.277606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.378120 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.378081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pp4\" (UniqueName: \"kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.378316 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.378138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.378316 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.378201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.380814 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.380782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.380984 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.380844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.386781 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.386749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pp4\" (UniqueName: \"kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4\") pod \"maas-controller-7dbf7685b-xc2qh\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") " pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.486986 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.486879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:38.761496 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.761469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:38.763835 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:06:38.763802    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16472800_a52a_4ed2_b0b1_20802369d5ce.slice/crio-21817d7c791859b65292f9bbcb53baf0087f0bb67f3e65684523af1a913c0809 WatchSource:0}: Error finding container 21817d7c791859b65292f9bbcb53baf0087f0bb67f3e65684523af1a913c0809: Status 404 returned error can't find the container with id 21817d7c791859b65292f9bbcb53baf0087f0bb67f3e65684523af1a913c0809
Sep 16 01:06:38.765322 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:38.765299    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:06:39.562998 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:39.562964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" event={"ID":"16472800-a52a-4ed2-b0b1-20802369d5ce","Type":"ContainerStarted","Data":"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"}
Sep 16 01:06:39.563373 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:39.563019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" event={"ID":"16472800-a52a-4ed2-b0b1-20802369d5ce","Type":"ContainerStarted","Data":"21817d7c791859b65292f9bbcb53baf0087f0bb67f3e65684523af1a913c0809"}
Sep 16 01:06:39.563373 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:39.563140    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:39.581534 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:39.581483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" podStartSLOduration=1.2795104099999999 podStartE2EDuration="1.581470691s" totalImagesPullingTime="301.960281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:06:39.580496723 +0000 UTC m=+1969.742007009" watchObservedRunningTime="2026-09-16 01:06:39.581470691 +0000 UTC m=+1969.742980997"
Sep 16 01:06:42.327620 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:42.327538    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:06:50.570486 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:50.570455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:57.923794 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:57.923756    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:57.924395 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:57.924027    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" podUID="16472800-a52a-4ed2-b0b1-20802369d5ce" containerName="manager" containerID="cri-o://b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36" gracePeriod=30
Sep 16 01:06:58.178595 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.178522    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:58.248318 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.248289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert\") pod \"16472800-a52a-4ed2-b0b1-20802369d5ce\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") "
Sep 16 01:06:58.248483 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.248326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs\") pod \"16472800-a52a-4ed2-b0b1-20802369d5ce\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") "
Sep 16 01:06:58.248483 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.248414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2pp4\" (UniqueName: \"kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4\") pod \"16472800-a52a-4ed2-b0b1-20802369d5ce\" (UID: \"16472800-a52a-4ed2-b0b1-20802369d5ce\") "
Sep 16 01:06:58.250675 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.250639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "16472800-a52a-4ed2-b0b1-20802369d5ce" (UID: "16472800-a52a-4ed2-b0b1-20802369d5ce"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:58.250820 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.250669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16472800-a52a-4ed2-b0b1-20802369d5ce" (UID: "16472800-a52a-4ed2-b0b1-20802369d5ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 01:06:58.250820 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.250781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4" (OuterVolumeSpecName: "kube-api-access-q2pp4") pod "16472800-a52a-4ed2-b0b1-20802369d5ce" (UID: "16472800-a52a-4ed2-b0b1-20802369d5ce"). InnerVolumeSpecName "kube-api-access-q2pp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 01:06:58.349391 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.349367    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pp4\" (UniqueName: \"kubernetes.io/projected/16472800-a52a-4ed2-b0b1-20802369d5ce-kube-api-access-q2pp4\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:06:58.349391 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.349392    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-webhook-cert\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:06:58.349555 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.349401    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16472800-a52a-4ed2-b0b1-20802369d5ce-metrics-certs\") on node \"ip-10-0-134-152.ec2.internal\" DevicePath \"\""
Sep 16 01:06:58.670832 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.670791    2577 generic.go:356] "Generic (PLEG): container finished" podID="16472800-a52a-4ed2-b0b1-20802369d5ce" containerID="b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36" exitCode=0
Sep 16 01:06:58.671095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.670862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-xc2qh"
Sep 16 01:06:58.671095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.670881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" event={"ID":"16472800-a52a-4ed2-b0b1-20802369d5ce","Type":"ContainerDied","Data":"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"}
Sep 16 01:06:58.671095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.670943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-xc2qh" event={"ID":"16472800-a52a-4ed2-b0b1-20802369d5ce","Type":"ContainerDied","Data":"21817d7c791859b65292f9bbcb53baf0087f0bb67f3e65684523af1a913c0809"}
Sep 16 01:06:58.671095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.670987    2577 scope.go:117] "RemoveContainer" containerID="b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"
Sep 16 01:06:58.682666 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.682635    2577 scope.go:117] "RemoveContainer" containerID="b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"
Sep 16 01:06:58.683071 ip-10-0-134-152 kubenswrapper[2577]: E0916 01:06:58.683032    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36\": container with ID starting with b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36 not found: ID does not exist" containerID="b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"
Sep 16 01:06:58.683192 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.683065    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36"} err="failed to get container status \"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36\": rpc error: code = NotFound desc = could not find container \"b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36\": container with ID starting with b075cbd1787941975fb71cd713090fd2476a13b8edfc118525f8f3459071fc36 not found: ID does not exist"
Sep 16 01:06:58.690379 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.690275    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:58.692067 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:58.692044    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-xc2qh"]
Sep 16 01:06:59.536899 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.536857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dbf7685b-6hq9d"]
Sep 16 01:06:59.537310 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537294    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16472800-a52a-4ed2-b0b1-20802369d5ce" containerName="manager"
Sep 16 01:06:59.537310 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16472800-a52a-4ed2-b0b1-20802369d5ce" containerName="manager"
Sep 16 01:06:59.537388 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537321    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:59.537388 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537327    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:59.537460 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537396    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52cf260b-293a-4718-88f2-f8790dc97434" containerName="istio-proxy"
Sep 16 01:06:59.537460 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537405    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e688e66c-2e95-4b5d-8448-f1e0888efaa7" containerName="istio-proxy"
Sep 16 01:06:59.537460 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.537412    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16472800-a52a-4ed2-b0b1-20802369d5ce" containerName="manager"
Sep 16 01:06:59.541830 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.541814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.545578 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.545551    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk8jt\""
Sep 16 01:06:59.545732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.545562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 01:06:59.545732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.545563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 01:06:59.545732 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.545563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 01:06:59.549329 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.549266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-6hq9d"]
Sep 16 01:06:59.660570 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.660529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2p5\" (UniqueName: \"kubernetes.io/projected/5e91509c-6065-4d08-a121-7ab08f3edfe3-kube-api-access-bk2p5\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.660570 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.660576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-webhook-cert\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.660827 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.660595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-metrics-certs\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.762168 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.762127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2p5\" (UniqueName: \"kubernetes.io/projected/5e91509c-6065-4d08-a121-7ab08f3edfe3-kube-api-access-bk2p5\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.762362 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.762192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-webhook-cert\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.762362 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.762214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-metrics-certs\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.765052 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.765023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-webhook-cert\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.765177 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.765070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e91509c-6065-4d08-a121-7ab08f3edfe3-metrics-certs\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.770303 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.770280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2p5\" (UniqueName: \"kubernetes.io/projected/5e91509c-6065-4d08-a121-7ab08f3edfe3-kube-api-access-bk2p5\") pod \"maas-controller-7dbf7685b-6hq9d\" (UID: \"5e91509c-6065-4d08-a121-7ab08f3edfe3\") " pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:06:59.853393 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:06:59.853304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:07:00.127942 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.127912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf7685b-6hq9d"]
Sep 16 01:07:00.129940 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:07:00.129908    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e91509c_6065_4d08_a121_7ab08f3edfe3.slice/crio-5f290ff2d793c13019cdda9672d057c16dfaabef2739283c023fca040e1ea074 WatchSource:0}: Error finding container 5f290ff2d793c13019cdda9672d057c16dfaabef2739283c023fca040e1ea074: Status 404 returned error can't find the container with id 5f290ff2d793c13019cdda9672d057c16dfaabef2739283c023fca040e1ea074
Sep 16 01:07:00.334493 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.334457    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16472800-a52a-4ed2-b0b1-20802369d5ce" path="/var/lib/kubelet/pods/16472800-a52a-4ed2-b0b1-20802369d5ce/volumes"
Sep 16 01:07:00.685131 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.685099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-6hq9d" event={"ID":"5e91509c-6065-4d08-a121-7ab08f3edfe3","Type":"ContainerStarted","Data":"bc44eb8a0fed6a49f8a729670a28e5338bec4fea003a5a7dff221424a87ab4a2"}
Sep 16 01:07:00.685531 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.685158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf7685b-6hq9d" event={"ID":"5e91509c-6065-4d08-a121-7ab08f3edfe3","Type":"ContainerStarted","Data":"5f290ff2d793c13019cdda9672d057c16dfaabef2739283c023fca040e1ea074"}
Sep 16 01:07:00.685531 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.685229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:07:00.703567 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:00.703505    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dbf7685b-6hq9d" podStartSLOduration=1.314967524 podStartE2EDuration="1.703488153s" totalImagesPullingTime="388.520629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:07:00.701465395 +0000 UTC m=+1990.862975667" watchObservedRunningTime="2026-09-16 01:07:00.703488153 +0000 UTC m=+1990.864998469"
Sep 16 01:07:11.692234 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:11.692202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dbf7685b-6hq9d"
Sep 16 01:07:17.327995 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:17.327936    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:07:56.328084 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:07:56.328048    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:08:20.330190 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:08:20.330099    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:08:50.491620 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:08:50.491580    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:08:50.503037 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:08:50.503017    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:09:25.328045 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:09:25.328008    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:09:34.327567 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:09:34.327531    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:10:42.328145 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:10:42.328102    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:11:00.330043 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:11:00.330007    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:11:58.328243 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:11:58.328204    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:12:02.328326 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:12:02.328289    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:13:08.327576 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:13:08.327539    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:13:19.328317 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:13:19.328281    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:13:50.524552 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:13:50.524523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:13:50.536654 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:13:50.536628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:14:23.328220 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:23.328138    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-6cd864bc95-l9q92" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:14:28.328245 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:28.328204    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-h5j6g" secret="" err="secret \"payload-processing-dockercfg-sgjrw\" not found"
Sep 16 01:14:37.703436 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.703391    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jd2dw/must-gather-mtksj"]
Sep 16 01:14:37.706982 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.706936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.709365 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.709342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jd2dw\"/\"default-dockercfg-v76dc\""
Sep 16 01:14:37.709580 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.709562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jd2dw\"/\"kube-root-ca.crt\""
Sep 16 01:14:37.709678 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.709607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jd2dw\"/\"openshift-service-ca.crt\""
Sep 16 01:14:37.722510 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.722485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/must-gather-mtksj"]
Sep 16 01:14:37.831757 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.831726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2phx\" (UniqueName: \"kubernetes.io/projected/987ffa0f-5aca-4c69-80f5-017db20d66fd-kube-api-access-b2phx\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.831927 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.831765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/987ffa0f-5aca-4c69-80f5-017db20d66fd-must-gather-output\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.933138 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.933108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2phx\" (UniqueName: \"kubernetes.io/projected/987ffa0f-5aca-4c69-80f5-017db20d66fd-kube-api-access-b2phx\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.933297 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.933146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/987ffa0f-5aca-4c69-80f5-017db20d66fd-must-gather-output\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.933415 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.933401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/987ffa0f-5aca-4c69-80f5-017db20d66fd-must-gather-output\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:37.941449 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:37.941417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2phx\" (UniqueName: \"kubernetes.io/projected/987ffa0f-5aca-4c69-80f5-017db20d66fd-kube-api-access-b2phx\") pod \"must-gather-mtksj\" (UID: \"987ffa0f-5aca-4c69-80f5-017db20d66fd\") " pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:38.015988 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:38.015961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/must-gather-mtksj"
Sep 16 01:14:38.294095 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:38.294066    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/must-gather-mtksj"]
Sep 16 01:14:38.296015 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:14:38.295987    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987ffa0f_5aca_4c69_80f5_017db20d66fd.slice/crio-bfa5af0999615f15deaa46e77d1ca231cea67814ec31138894808601788b288d WatchSource:0}: Error finding container bfa5af0999615f15deaa46e77d1ca231cea67814ec31138894808601788b288d: Status 404 returned error can't find the container with id bfa5af0999615f15deaa46e77d1ca231cea67814ec31138894808601788b288d
Sep 16 01:14:38.298213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:38.298195    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 01:14:39.046683 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:39.046650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/must-gather-mtksj" event={"ID":"987ffa0f-5aca-4c69-80f5-017db20d66fd","Type":"ContainerStarted","Data":"bfa5af0999615f15deaa46e77d1ca231cea67814ec31138894808601788b288d"}
Sep 16 01:14:40.057022 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:40.056985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/must-gather-mtksj" event={"ID":"987ffa0f-5aca-4c69-80f5-017db20d66fd","Type":"ContainerStarted","Data":"8bd0aab6cf464b71d8781e55c33c6bab0ced4815995b601d37587aefe59f01d0"}
Sep 16 01:14:40.057455 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:40.057035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/must-gather-mtksj" event={"ID":"987ffa0f-5aca-4c69-80f5-017db20d66fd","Type":"ContainerStarted","Data":"d32836d9bdbc92ca07a30471836404580b7f50e074ffb66784c50eba00fe7450"}
Sep 16 01:14:40.073097 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:40.073033    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jd2dw/must-gather-mtksj" podStartSLOduration=1.938788177 podStartE2EDuration="3.073010609s" totalImagesPullingTime="1.134222432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:40.072289593 +0000 UTC m=+2450.233799865" watchObservedRunningTime="2026-09-16 01:14:40.073010609 +0000 UTC m=+2450.234520898"
Sep 16 01:14:41.049289 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:41.049266    2577 ???:1] "http: TLS handshake error from 10.0.131.194:38034: EOF"
Sep 16 01:14:41.056675 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:41.056645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z2xsn_6539572a-35ca-4938-a691-21ff110ed047/global-pull-secret-syncer/0.log"
Sep 16 01:14:41.185881 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:41.185847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xvptl_995499f7-f75d-445e-8ce3-a8b0aa5c860d/konnectivity-agent/0.log"
Sep 16 01:14:41.227305 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:41.227277    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-152.ec2.internal_3d95a97eebf9880fb8b18802bb04714a/haproxy/0.log"
Sep 16 01:14:46.076998 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:46.076970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-pbbrm_f66545bd-d6b9-4daf-af99-260078e24d84/manager/0.log"
Sep 16 01:14:46.103419 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:46.103331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-dldsr_bd0ab4c9-5929-4726-82d5-b32d1ce186f5/manager/0.log"
Sep 16 01:14:46.282617 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:46.282584    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7wlls_e45e50df-6c82-4f5b-8d9f-a59596e928a6/manager/0.log"
Sep 16 01:14:47.758436 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:47.758407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-fkmdm_cc191bfb-a7ab-4ce4-994b-518fa5b494df/cluster-monitoring-operator/0.log"
Sep 16 01:14:47.932469 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:47.932435    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gb65z_73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c/node-exporter/0.log"
Sep 16 01:14:47.959600 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:47.959563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gb65z_73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c/kube-rbac-proxy/0.log"
Sep 16 01:14:47.983184 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:47.983102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gb65z_73b7eeab-06e7-4f7c-b7ce-0a65f0edc78c/init-textfile/0.log"
Sep 16 01:14:50.086940 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:50.086885    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-mb5h7_665ca7f3-d81e-4286-b5ce-1074c872f927/networking-console-plugin/0.log"
Sep 16 01:14:50.673059 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:50.673027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/1.log"
Sep 16 01:14:50.679575 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:50.679546    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hzw72_4305e130-063f-4318-a07c-7a6cb748c1ac/console-operator/2.log"
Sep 16 01:14:51.003032 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.002908    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"]
Sep 16 01:14:51.009077 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.009045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.015086 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.015054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"]
Sep 16 01:14:51.177489 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.177436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-lib-modules\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.178123 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.178093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9db\" (UniqueName: \"kubernetes.io/projected/65e0d43a-d227-400a-8da5-60af39312b78-kube-api-access-wm9db\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.178224 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.178168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-sys\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.178224 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.178210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-proc\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.178348 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.178275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-podres\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.279125 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-sys\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.279428 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-proc\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.279608 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-proc\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.279784 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-podres\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.279936 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-lib-modules\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.280069 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.280047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-lib-modules\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.280213 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-sys\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.280301 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.280099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9db\" (UniqueName: \"kubernetes.io/projected/65e0d43a-d227-400a-8da5-60af39312b78-kube-api-access-wm9db\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.280483 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.279696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65e0d43a-d227-400a-8da5-60af39312b78-podres\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.289143 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.289111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9db\" (UniqueName: \"kubernetes.io/projected/65e0d43a-d227-400a-8da5-60af39312b78-kube-api-access-wm9db\") pod \"perf-node-gather-daemonset-dhpjv\" (UID: \"65e0d43a-d227-400a-8da5-60af39312b78\") " pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.327532 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.327494    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-ldwk9_412a03f2-398c-4be0-a7f6-0765c7da8310/download-server/0.log"
Sep 16 01:14:51.330458 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.330431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:51.631516 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:51.631476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"]
Sep 16 01:14:51.634821 ip-10-0-134-152 kubenswrapper[2577]: W0916 01:14:51.634774    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65e0d43a_d227_400a_8da5_60af39312b78.slice/crio-21803f5a6afd6a20de14c5cbd0c9726290f766a9a972ca13c60e931e2bf4d3a9 WatchSource:0}: Error finding container 21803f5a6afd6a20de14c5cbd0c9726290f766a9a972ca13c60e931e2bf4d3a9: Status 404 returned error can't find the container with id 21803f5a6afd6a20de14c5cbd0c9726290f766a9a972ca13c60e931e2bf4d3a9
Sep 16 01:14:52.040083 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:52.040055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-q9xwt_e990c269-b651-4030-90df-1b8c31a5468d/volume-data-source-validator/0.log"
Sep 16 01:14:52.154624 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:52.154582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv" event={"ID":"65e0d43a-d227-400a-8da5-60af39312b78","Type":"ContainerStarted","Data":"1b5a7e0284322895708f704ac6687809115391edfb58e8bdcadfa765e7becf21"}
Sep 16 01:14:52.154819 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:52.154649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv" event={"ID":"65e0d43a-d227-400a-8da5-60af39312b78","Type":"ContainerStarted","Data":"21803f5a6afd6a20de14c5cbd0c9726290f766a9a972ca13c60e931e2bf4d3a9"}
Sep 16 01:14:52.155005 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:52.154977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:52.173985 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:52.171643    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv" podStartSLOduration=2.171621631 podStartE2EDuration="2.171621631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:14:52.170752963 +0000 UTC m=+2462.332263236" watchObservedRunningTime="2026-09-16 01:14:52.171621631 +0000 UTC m=+2462.333131904"
Sep 16 01:14:53.296365 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:53.296333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qctk_b85b14f1-369a-4c29-9c33-91420ee78f85/dns/0.log"
Sep 16 01:14:53.317300 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:53.317263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qctk_b85b14f1-369a-4c29-9c33-91420ee78f85/kube-rbac-proxy/0.log"
Sep 16 01:14:53.429916 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:53.429884    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzgkz_3226ed81-5d7e-4d58-837d-99ede28bbcbf/dns-node-resolver/0.log"
Sep 16 01:14:54.052184 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:54.052142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztd8m_9b83e392-9d9a-477b-9d05-cd1b5d124150/node-ca/0.log"
Sep 16 01:14:55.674830 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.674802    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bb4c8f6-l2fdx_306f1800-c75b-4910-8922-f5aca47f70f4/kube-auth-proxy/0.log"
Sep 16 01:14:55.705310 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.705252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/0.log"
Sep 16 01:14:55.709642 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.709608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/1.log"
Sep 16 01:14:55.732575 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.732526    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-l9q92_f0ba643b-242e-49bb-9a71-1746afd71e33/payload-pre-processing/0.log"
Sep 16 01:14:55.765304 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.765275    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:14:55.794379 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:55.794353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5475d44bcb-9c4pt_864ca34e-436f-4a8d-a0b0-d3349b46a4ef/router/0.log"
Sep 16 01:14:56.650704 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:56.650653    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bqqnb_2a739a79-a67d-4add-abca-aa30d2f17687/serve-healthcheck-canary/0.log"
Sep 16 01:14:57.158635 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:57.158604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-zxsw2_a633fe59-ba7f-4981-a1a0-0533327434e8/insights-operator/0.log"
Sep 16 01:14:57.159334 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:57.159315    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-zxsw2_a633fe59-ba7f-4981-a1a0-0533327434e8/insights-operator/1.log"
Sep 16 01:14:57.181692 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:57.181661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdpd7_a0227a00-e4a5-433a-a5b7-c7020fdd10f5/kube-rbac-proxy/0.log"
Sep 16 01:14:57.203668 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:57.203639    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdpd7_a0227a00-e4a5-433a-a5b7-c7020fdd10f5/exporter/0.log"
Sep 16 01:14:57.226394 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:57.226363    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hdpd7_a0227a00-e4a5-433a-a5b7-c7020fdd10f5/extractor/0.log"
Sep 16 01:14:58.174587 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:58.174555    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jd2dw/perf-node-gather-daemonset-dhpjv"
Sep 16 01:14:59.465879 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:59.465847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-57d6998586-rtxvr_556b217d-ad1d-4731-a50f-72a315d721d4/manager/0.log"
Sep 16 01:14:59.491171 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:59.491141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r7gtb_4946dd9c-c8b5-4e77-a870-12138bdb5102/manager/0.log"
Sep 16 01:14:59.818118 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:59.818081    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v6wsp_ea02c116-d193-437b-9841-e97747a744e6/manager/0.log"
Sep 16 01:14:59.880707 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:14:59.880668    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf7685b-6hq9d_5e91509c-6065-4d08-a121-7ab08f3edfe3/manager/0.log"
Sep 16 01:15:00.277640 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:00.277602    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d645f7ff-zt62q_b353ecb1-0c72-4ca4-b1dd-065ae24b1836/manager/0.log"
Sep 16 01:15:00.303075 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:00.303043    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d645f7ff-zt62q_b353ecb1-0c72-4ca4-b1dd-065ae24b1836/copy-manifests/0.log"
Sep 16 01:15:02.668546 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.668516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bb4c8f6-l2fdx_306f1800-c75b-4910-8922-f5aca47f70f4/kube-auth-proxy/0.log"
Sep 16 01:15:02.688748 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.688715    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/0.log"
Sep 16 01:15:02.691278 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.691253    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/1.log"
Sep 16 01:15:02.704262 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.704232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-l9q92_f0ba643b-242e-49bb-9a71-1746afd71e33/payload-pre-processing/0.log"
Sep 16 01:15:02.718719 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.718689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:15:02.743318 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:02.743290    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5475d44bcb-9c4pt_864ca34e-436f-4a8d-a0b0-d3349b46a4ef/router/0.log"
Sep 16 01:15:04.981652 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:04.981582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bb4c8f6-l2fdx_306f1800-c75b-4910-8922-f5aca47f70f4/kube-auth-proxy/0.log"
Sep 16 01:15:05.006025 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:05.005988    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/0.log"
Sep 16 01:15:05.008931 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:05.008897    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6m5tb_8a77ed0e-564e-463e-947e-d6501435fbb1/istio-proxy/1.log"
Sep 16 01:15:05.021078 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:05.021046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cd864bc95-l9q92_f0ba643b-242e-49bb-9a71-1746afd71e33/payload-pre-processing/0.log"
Sep 16 01:15:05.037658 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:05.037586    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-h5j6g_d7344110-1e8e-40f1-b9e8-c490a1b44d1b/payload-processing/0.log"
Sep 16 01:15:05.063502 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:05.063472    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5475d44bcb-9c4pt_864ca34e-436f-4a8d-a0b0-d3349b46a4ef/router/0.log"
Sep 16 01:15:06.989274 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:06.989225    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-45fpt_dcbeaf8f-1b56-4ccb-9254-c580e53589d8/migrator/0.log"
Sep 16 01:15:07.017101 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:07.017072    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-45fpt_dcbeaf8f-1b56-4ccb-9254-c580e53589d8/graceful-termination/0.log"
Sep 16 01:15:07.380326 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:07.380281    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5dc8v_7cde447c-7ce6-4fa6-98a2-4853d9fd5179/kube-storage-version-migrator-operator/1.log"
Sep 16 01:15:07.381106 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:07.381085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5dc8v_7cde447c-7ce6-4fa6-98a2-4853d9fd5179/kube-storage-version-migrator-operator/0.log"
Sep 16 01:15:08.371051 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.371018    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/kube-multus-additional-cni-plugins/0.log"
Sep 16 01:15:08.394121 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.394091    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/egress-router-binary-copy/0.log"
Sep 16 01:15:08.417259 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.417228    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/cni-plugins/0.log"
Sep 16 01:15:08.439803 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.439771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/bond-cni-plugin/0.log"
Sep 16 01:15:08.461424 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.461396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/routeoverride-cni/0.log"
Sep 16 01:15:08.486049 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.486025    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/whereabouts-cni-bincopy/0.log"
Sep 16 01:15:08.508722 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.508689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gkht_0a72feb0-e226-4441-b535-e80f772ad304/whereabouts-cni/0.log"
Sep 16 01:15:08.997548 ip-10-0-134-152 kubenswrapper[2577]: I0916 01:15:08.997497    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n9fp7_9d763a7a-b3b9-4600-940b-cfcc45815f0b/kube-multus/0.log"