Sep 09 18:31:32.369910 ip-10-0-135-104 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 18:31:32.799116 ip-10-0-135-104 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:32.799116 ip-10-0-135-104 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 18:31:32.799116 ip-10-0-135-104 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:32.799116 ip-10-0-135-104 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 18:31:32.799116 ip-10-0-135-104 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:32.801572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.801486    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 18:31:32.806209 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806194    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.806209 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806209    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806212    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806215    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806221    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806225    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806229    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806232    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806234    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806237    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806240    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806243    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806246    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806249    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806252    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806255    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806257    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806260    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806262    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806265    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806271    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.806275 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806274    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806276    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806279    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806282    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806285    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806288    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806290    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806293    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806295    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806297    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806300    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806303    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806305    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806308    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806310    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806313    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806315    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806318    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806321    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806324    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.806765 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806327    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806329    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806332    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806334    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806336    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806339    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806341    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806344    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806346    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806348    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806351    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806353    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806356    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806359    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806362    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806365    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806368    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806370    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806373    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806375    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.807282 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806378    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806380    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806383    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806385    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806387    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806390    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806392    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806395    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806397    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806400    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806402    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806405    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806407    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806410    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806412    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806415    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806418    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806421    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806423    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.807889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806426    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806428    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806430    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806433    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806435    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806438    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806441    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806447    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806450    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806453    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806455    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806458    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806460    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806463    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806465    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806468    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806470    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806473    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.806477    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.808382 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807162    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807169    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807172    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807175    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807178    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807180    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807183    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807185    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807188    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807191    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807194    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807197    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807199    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807202    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807204    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807207    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807210    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807212    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807214    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807217    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.808865 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807221    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807224    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807227    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807229    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807232    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807234    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807237    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807240    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807244    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807247    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807249    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807252    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807254    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807257    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807260    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807262    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807264    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807267    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807269    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.809367 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807272    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807274    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807276    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807279    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807283    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807286    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807288    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807291    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807294    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807296    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807299    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807302    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807304    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807307    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807310    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807312    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807315    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807317    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807320    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807322    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.809858 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807325    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807328    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807330    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807333    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807335    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807338    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807340    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807343    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807345    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807347    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807350    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807352    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807354    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807357    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807359    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807362    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807366    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807369    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807372    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807374    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.810357 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807377    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807379    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807382    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807386    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807388    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807391    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807393    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807397    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807401    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807404    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807406    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807409    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807411    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807414    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807416    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807420    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807423    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807426    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807429    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.810864 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807432    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807854    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807860    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807863    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807866    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807869    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807872    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807875    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807880    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807883    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807886    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807889    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807892    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807895    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807898    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807902    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807905    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807907    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807910    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807913    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.811349 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807916    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807919    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807921    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807924    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807926    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807929    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807931    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807934    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807936    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807939    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807941    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807943    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807946    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807948    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807951    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807953    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807957    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807960    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807964    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807967    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.811856 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807970    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807973    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807975    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807978    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807980    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807983    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807985    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807988    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807992    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807994    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807997    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.807999    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808002    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808005    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808007    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808010    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808012    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808015    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808017    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.812345 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808020    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808022    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808024    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808027    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808029    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808031    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808034    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808036    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808039    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808041    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808045    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808048    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808052    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808056    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808060    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808063    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808066    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808069    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808071    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.812849 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808074    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808076    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808079    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808081    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808084    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808087    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808089    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808091    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808094    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808097    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808099    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808102    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808104    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808107    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808109    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808112    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808114    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808117    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808119    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808122    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.813317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808124    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808126    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808191    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808224    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808231    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808236    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808241    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808244    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808248    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808253    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808256    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808259    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808262    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808265    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808268    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808271    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808274    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808277    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808280    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808282    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808286    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808290    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808293    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808296    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 18:31:32.813815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808299    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808302    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808305    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808308    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808311    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808313    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808316    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808319    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808322    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808325    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808329    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808332    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808335    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808337    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808343    2581 flags.go:64] FLAG: --enable-server="true"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808346    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808350    2581 flags.go:64] FLAG: --event-burst="100"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808353    2581 flags.go:64] FLAG: --event-qps="50"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808356    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808358    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808361    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808365    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808368    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808371    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808374    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 09 18:31:32.814422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808377    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808380    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808383    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808386    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808388    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808392    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808395    2581 flags.go:64] FLAG: --feature-gates=""
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808399    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808402    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808405    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808408    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808410    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808413    2581 flags.go:64] FLAG: --help="false"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808416    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-104.ec2.internal"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808419    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808422    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808425    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808428    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808431    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808434    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808437    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808441    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808443    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 18:31:32.815030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808446    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808450    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808452    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808455    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808459    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808462    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808465    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808467    2581 flags.go:64] FLAG: --lock-file=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808470    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808473    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808476    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808480    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808483    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808486    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808489    2581 flags.go:64] FLAG: --logging-format="text"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808492    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808495    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808498    2581 flags.go:64] FLAG: --manifest-url=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808500    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808504    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808507    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808511    2581 flags.go:64] FLAG: --max-pods="110"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808513    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808516    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808519    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 18:31:32.815621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808521    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808524    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808527    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808530    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808536    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808541    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808544    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808547    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808550    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808555    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808558    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808561    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808567    2581 flags.go:64] FLAG: --port="10250"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808570    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808573    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06778e4f4f2d45d7f"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808576    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808578    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808582    2581 flags.go:64] FLAG: --register-node="true"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808585    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808588    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808591    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808594    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808597    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808601    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808604    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 18:31:32.816235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808620    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808624    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808626    2581 flags.go:64] FLAG: --runonce="false"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808629    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808632    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808635    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808638    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808641    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808644    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808647    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808650    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808653    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808658    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808661    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808663    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808666    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808669    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808672    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808678    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808681    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808684    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808688    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808691    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808693    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808696    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 18:31:32.816848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808699    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808702    2581 flags.go:64] FLAG: --v="2"
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808705    2581 flags.go:64] FLAG: --version="false"
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808709    2581 flags.go:64] FLAG: --vmodule=""
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808713    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.808716    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808861    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808865    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808868    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808871    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808874    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808877    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808880    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808883    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808886    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808888    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808901    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808905    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808908    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808912    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808914    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.817460 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808917    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808920    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808923    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808926    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808929    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808933    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808936    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808938    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808941    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808943    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808946    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808948    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808951    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808953    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808956    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808958    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808960    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808963    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808966    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808969    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.818104 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808971    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808974    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808976    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808978    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808981    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808983    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808987    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808990    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808993    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808996    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.808999    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809002    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809004    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809007    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809009    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809012    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809014    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809019    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809022    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.818587 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809024    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809026    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809029    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809031    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809034    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809036    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809039    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809041    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809044    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809048    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809051    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809053    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809056    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809059    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809061    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809064    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809066    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809069    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809071    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.819052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809074    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809076    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809079    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809081    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809084    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809087    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809089    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809093    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809096    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809099    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809102    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809105    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809108    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809110    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809113    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809115    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809118    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809121    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809123    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809126    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.819511 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809128    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809130    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809133    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809136    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809138    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.809140    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.820023 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.809825    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:32.827593 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.827566    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 18:31:32.827593 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.827588    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.827630    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.827650    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827730    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827738    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827742    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827749    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827756    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827761    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827765    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827769    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827774    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827778    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827782    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827786    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827790    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.827780 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827794    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827800    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827804    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827809    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827813    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827818    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827822    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827826    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827830    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827835    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827838    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827842    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827847    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827850    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827854    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827858    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827863    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827867    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827871    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827875    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.828497 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827879    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827883    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827890    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827895    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827900    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827904    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827908    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827912    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827916    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827920    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827924    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827928    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827932    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827937    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827942    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827946    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827950    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827954    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827959    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.829297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827963    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827968    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827972    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827976    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827980    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827984    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827988    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827992    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.827999    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828004    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828009    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828013    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828018    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828022    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828026    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828031    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828035    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828040    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828044    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828049    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.830065 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828053    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828057    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828061    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828065    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828069    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828073    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828077    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828083    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828087    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828091    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828096    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828100    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828104    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828108    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828112    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828116    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828120    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828123    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828127    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828131    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.830564 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828136    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828139    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828144    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828147    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828152    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828155    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828160    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.828167    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828346    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828356    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828361    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828365    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828369    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828374    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828378    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:32.831244 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828382    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828386    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828390    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828394    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828399    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828412    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828419    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828424    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828429    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828434    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828438    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828443    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828447    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828451    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828455    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828460    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828464    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828469    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828473    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:32.831903 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828477    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828481    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828485    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828489    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828493    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828497    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828501    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828505    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828509    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828513    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828517    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828520    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828525    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828529    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828535    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828540    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828544    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828548    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828553    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828559    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:32.832435 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828563    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828567    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828573    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828579    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828584    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828588    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828593    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828597    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828601    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828605    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828629    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828634    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828638    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828642    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828647    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828651    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828655    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828659    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828663    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:32.833083 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828667    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828672    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828675    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828680    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828684    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828688    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828692    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828696    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828700    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828705    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828710    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828714    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828718    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828724    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828728    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828732    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828736    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828740    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828745    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828749    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:32.833540 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828753    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828757    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828761    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828765    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828769    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828773    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828777    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828781    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828785    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828789    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828793    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828797    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828801    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:32.828806    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.828814    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:32.834147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.828971    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 18:31:32.834516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.832663    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 18:31:32.834516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.833902    2581 server.go:1019] "Starting client certificate rotation"
Sep 09 18:31:32.834516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.834002    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 18:31:32.834516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.834040    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 18:31:32.836477 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.836459    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 18:31:32.839661 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.839598    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 18:31:32.849939 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.849918    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 09 18:31:32.852901 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.852886    2581 log.go:25] "Validated CRI v1 image API"
Sep 09 18:31:32.853708 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.853694    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 18:31:32.857376 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.857356    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a8aa8255-7298-4668-9fe9-3027a8014101:/dev/nvme0n1p4 e0c6c1d6-4607-4b2d-8e5d-acc55b70bdeb:/dev/nvme0n1p3]
Sep 09 18:31:32.857450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.857375    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 18:31:32.863646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.863219    2581 manager.go:216] Machine: {Timestamp:2026-09-09 18:31:32.861994001 +0000 UTC m=+0.367697052 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095873 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26898d0211b81144d58c982edbd754 SystemUUID:ec26898d-0211-b811-44d5-8c982edbd754 BootID:d03c0708-d94b-4925-a852-21099bd594b8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:fc:3e:73:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:fc:3e:73:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:4d:e9:de:5c:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 18:31:32.863646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.863641    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 18:31:32.863811 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.863760    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 18:31:32.867920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.867735    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 18:31:32.868107 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.867927    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 18:31:32.868466 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.868448    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 18:31:32.868994 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.868981    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 18:31:32.869032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.868998    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 18:31:32.869032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.869007    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 18:31:32.869032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.869021    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 18:31:32.870430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.870419    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 18:31:32.874575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874564    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 18:31:32.874710 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874700    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 18:31:32.874747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874729    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 18:31:32.874747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874739    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 18:31:32.874800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874757    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 18:31:32.874800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874768    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 09 18:31:32.874800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.874777    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 18:31:32.875711 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.875699    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 18:31:32.875768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.875717    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 18:31:32.878400 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.878384    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 18:31:32.881035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881019    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 18:31:32.881090 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881050    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 18:31:32.881226 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881216    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 18:31:32.881259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881231    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 18:31:32.881259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881238    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 18:31:32.881259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881244    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 18:31:32.881259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881251    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 18:31:32.881259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881256    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 18:31:32.881904 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881894    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 18:31:32.881904 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881904    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 18:31:32.881972 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881910    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 18:31:32.881972 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881916    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 18:31:32.881972 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881924    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 18:31:32.881972 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.881932    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 18:31:32.882801 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.882790    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 18:31:32.882801 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.882799    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 18:31:32.885728 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.885703    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 18:31:32.885728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.885681    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 18:31:32.885728 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.885726    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 18:31:32.886727 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.886715    2581 server.go:1268] "Started kubelet"
Sep 09 18:31:32.886863 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.886836    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 18:31:32.886923 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.886851    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 18:31:32.886971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.886956    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 18:31:32.887554 ip-10-0-135-104 systemd[1]: Started Kubernetes Kubelet.
Sep 09 18:31:32.888504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.888269    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 18:31:32.888820 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.888808    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 18:31:32.892672 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.892652    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 18:31:32.893293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.893270    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 18:31:32.893823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.893779    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 18:31:32.893913 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.893846    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:32.894067 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.892982    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-104.ec2.internal.18d3baa48c40f62e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-104.ec2.internal,UID:ip-10-0-135-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-104.ec2.internal,},FirstTimestamp:2026-09-09 18:31:32.886689326 +0000 UTC m=+0.392392376,LastTimestamp:2026-09-09 18:31:32.886689326 +0000 UTC m=+0.392392376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-104.ec2.internal,}"
Sep 09 18:31:32.894524 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.894496    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 18:31:32.894524 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.894499    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 18:31:32.894652 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.894526    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 18:31:32.894652 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.894645    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 18:31:32.894723 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.894655    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 18:31:32.895894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.895788    2581 factory.go:55] Registering systemd factory
Sep 09 18:31:32.895894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.895802    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 09 18:31:32.896070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896057    2581 factory.go:153] Registering CRI-O factory
Sep 09 18:31:32.896111 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896071    2581 factory.go:223] Registration of the crio container factory successfully
Sep 09 18:31:32.896159 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896116    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 18:31:32.896159 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896133    2581 factory.go:103] Registering Raw factory
Sep 09 18:31:32.896159 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896149    2581 manager.go:1196] Started watching for new ooms in manager
Sep 09 18:31:32.896512 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.896499    2581 manager.go:349] Starting recovery of all containers
Sep 09 18:31:32.902170 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.902122    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 18:31:32.902422 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.902332    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 18:31:32.907109 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.907068    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 18:31:32.908075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.907961    2581 manager.go:354] Recovery completed
Sep 09 18:31:32.910369 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.910349    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bxxz4"
Sep 09 18:31:32.912321 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.912308    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:32.914834 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.914818    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:32.914911 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.914846    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:32.914911 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.914861    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:32.915413 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.915398    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 18:31:32.915413 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.915409    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 18:31:32.915527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.915424    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 18:31:32.915527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.915445    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:32.916155 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.916139    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:32.916235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.916167    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:32.916235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.916181    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:32.917003 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.916937    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-104.ec2.internal.18d3baa48dee6531  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-104.ec2.internal,UID:ip-10-0-135-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-104.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-104.ec2.internal,},FirstTimestamp:2026-09-09 18:31:32.914832689 +0000 UTC m=+0.420535740,LastTimestamp:2026-09-09 18:31:32.914832689 +0000 UTC m=+0.420535740,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-104.ec2.internal,}"
Sep 09 18:31:32.917581 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.917565    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bxxz4"
Sep 09 18:31:32.917745 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.917732    2581 policy_none.go:49] "None policy: Start"
Sep 09 18:31:32.917814 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.917749    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 18:31:32.917814 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.917759    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 18:31:32.919795 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.919783    2581 policy_none.go:47] "Start"
Sep 09 18:31:32.968198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968181    2581 manager.go:337] "Starting Device Plugin manager"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.968216    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968226    2581 server.go:87] "Starting device plugin registration server"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968502    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968512    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968605    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968718    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:32.968725    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.969162    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 18:31:32.982674 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:32.969217    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.003317 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.003298    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 18:31:33.003394 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.003348    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 18:31:33.003394 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.003369    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 18:31:33.003456 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.003402    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 18:31:33.005921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.005899    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:33.068934 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.068855    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:33.069832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.069813    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:33.069915 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.069855    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:33.069915 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.069868    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:33.069915 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.069893    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.079434 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.079417    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.079479 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.079440    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-104.ec2.internal\": node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.099418 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.099399    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.103484 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.103451    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"]
Sep 09 18:31:33.103593 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.103524    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:33.104884 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.104867    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:33.104984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.104902    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:33.104984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.104915    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:33.107045 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107029    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:33.107162 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.107210 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107180    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:33.107727 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107704    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:33.107727 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107730    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:33.107855 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107739    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:33.107855 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107774    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:33.107855 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107792    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:33.107855 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.107806    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:33.109893 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.109879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.109966 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.109904    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:33.110592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.110575    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:33.110692 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.110621    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:33.110692 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.110636    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:33.135168 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.135149    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-104.ec2.internal\" not found" node="ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.139603 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.139589    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-104.ec2.internal\" not found" node="ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.197202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.197175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.197202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.197204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.197345 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.197221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f59d360c99ceabc7a2e0508d7e2b003-config\") pod \"kube-apiserver-proxy-ip-10-0-135-104.ec2.internal\" (UID: \"9f59d360c99ceabc7a2e0508d7e2b003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.200290 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.200276    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.297793 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.297756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.299470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.297797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.299470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.298086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.299470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.298132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f59d360c99ceabc7a2e0508d7e2b003-config\") pod \"kube-apiserver-proxy-ip-10-0-135-104.ec2.internal\" (UID: \"9f59d360c99ceabc7a2e0508d7e2b003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.299470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.298160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed46da7d6b631c06d17627b8b432d0fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal\" (UID: \"ed46da7d6b631c06d17627b8b432d0fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.299470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.298176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f59d360c99ceabc7a2e0508d7e2b003-config\") pod \"kube-apiserver-proxy-ip-10-0-135-104.ec2.internal\" (UID: \"9f59d360c99ceabc7a2e0508d7e2b003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.300902 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.300883    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.401751 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.401679    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.437867 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.437839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.442995 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.442854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:33.501808 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.501776    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.602278 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.602249    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.702830 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.702776    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.771657 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.771627    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:33.803480 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.803460    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.833648 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.833621    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 18:31:33.833799 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.833779    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 18:31:33.833860 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.833807    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 18:31:33.904383 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:33.904356    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:33.921201 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.921166    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 18:26:34 +0000 UTC" deadline="2028-04-14 21:58:54.26855503 +0000 UTC"
Sep 09 18:31:33.921201 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:33.921194    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13995h27m20.347363517s"
Sep 09 18:31:34.004960 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:34.004889    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:34.104980 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:34.104950    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:34.182936 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:34.182905    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:34.205230 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:34.205194    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-104.ec2.internal\" not found"
Sep 09 18:31:34.219498 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:34.219468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f59d360c99ceabc7a2e0508d7e2b003.slice/crio-caf548ba52e8952659d7bbfeda6906de1b51682c56fc420fa851d2ba27b6ca6e WatchSource:0}: Error finding container caf548ba52e8952659d7bbfeda6906de1b51682c56fc420fa851d2ba27b6ca6e: Status 404 returned error can't find the container with id caf548ba52e8952659d7bbfeda6906de1b51682c56fc420fa851d2ba27b6ca6e
Sep 09 18:31:34.220003 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:34.219982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded46da7d6b631c06d17627b8b432d0fd.slice/crio-9e4cc729f3ab1309fd36a7ba240c0b1908c22a344d6f52c70db1307596ca9d70 WatchSource:0}: Error finding container 9e4cc729f3ab1309fd36a7ba240c0b1908c22a344d6f52c70db1307596ca9d70: Status 404 returned error can't find the container with id 9e4cc729f3ab1309fd36a7ba240c0b1908c22a344d6f52c70db1307596ca9d70
Sep 09 18:31:34.225010 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:34.224987    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:31:35.552594 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:35.552531    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:35.568645 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:35.568627    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:35.585941 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:35.585924    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 18:31:35.587085 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:35.587072    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal"
Sep 09 18:31:35.595275 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:35.595263    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 18:31:36.150385 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.150358    2581 apiserver.go:52] "Watching apiserver"
Sep 09 18:31:36.156764 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.156738    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 18:31:36.157068 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.157047    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qkdph","openshift-network-diagnostics/network-check-target-tp4fk","openshift-network-operator/iptables-alerter-qx427","openshift-ovn-kubernetes/ovnkube-node-d4vhl","kube-system/global-pull-secret-syncer-9hhwv","kube-system/konnectivity-agent-5284w","kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6p4k8","openshift-dns/node-resolver-h868k","openshift-image-registry/node-ca-2b2wp","openshift-multus/multus-7xzjw","openshift-multus/multus-additional-cni-plugins-7m9hr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal"]
Sep 09 18:31:36.160772 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.160754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.162932 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.162913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.163146 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.163039    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.163146 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.163075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs2rh\""
Sep 09 18:31:36.163275 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.163251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 18:31:36.163328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.163296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.165046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.165032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.166108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.165562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tjf6c\""
Sep 09 18:31:36.166108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.165869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 18:31:36.166108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.165942    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.166108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.165972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.167881 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.167557    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 18:31:36.168296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.168081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jx58\""
Sep 09 18:31:36.168296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.168140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.168296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.168224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.168296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.168228    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 18:31:36.169279 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.169259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.171425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.171407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 18:31:36.171531 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.171516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.171603 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.171582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:36.171692 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.171669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.171950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.171934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 18:31:36.172028 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.172000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.172085 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.172028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 18:31:36.172085 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.172066    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-879b5\""
Sep 09 18:31:36.172180 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.172084    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 18:31:36.173735 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.173718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.176017 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.175998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.176095 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.176062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 18:31:36.176095 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.176078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 18:31:36.176206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.176063    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xjcql\""
Sep 09 18:31:36.178068 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.178050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9l728\""
Sep 09 18:31:36.178179 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.178122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.178282 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.178269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.178383 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.178368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.179616 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-bin\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179689 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-script-lib\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179689 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-netns\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-iptables-alerter-script\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.179788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.179875 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-device-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.179875 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-node-log\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lv7\" (UniqueName: \"kubernetes.io/projected/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kube-api-access-c7lv7\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.179971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-systemd-units\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.179971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.179980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-log-socket\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovn-node-metrics-cert\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9648bc4f-28e2-4501-b9d7-f8d04caa8550-serviceca\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.180125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-host-slash\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.180125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-slash\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-systemd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-env-overrides\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-etc-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-ovn\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-config\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180343 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6dt\" (UniqueName: \"kubernetes.io/projected/c1ee21d1-a8ea-4e75-8542-8a37effeec09-kube-api-access-2q6dt\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-registration-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-netd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9648bc4f-28e2-4501-b9d7-f8d04caa8550-host\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkrn\" (UniqueName: \"kubernetes.io/projected/9648bc4f-28e2-4501-b9d7-f8d04caa8550-kube-api-access-9qkrn\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-plr25\""
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-socket-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-kubelet\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-sys-fs\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.180634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqsw\" (UniqueName: \"kubernetes.io/projected/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-kube-api-access-nwqsw\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.181327 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-var-lib-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.181327 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180692    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 18:31:36.181327 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.180777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.181327 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.181046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:36.181327 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.181160    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:36.184960 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.184945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.185050 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.185015    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:36.187461 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.187447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.189700 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.189683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 18:31:36.189891 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.189877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 18:31:36.189937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.189880    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 18:31:36.190084 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.190073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g5zzl\""
Sep 09 18:31:36.190141 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.190124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 18:31:36.190502 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.190483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.192767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.192636    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 18:31:36.192767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.192738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vgbjg\""
Sep 09 18:31:36.192767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.192758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 18:31:36.202017 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.202002    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l88wq"
Sep 09 18:31:36.207328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.207314    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l88wq"
Sep 09 18:31:36.269477 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.269457    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 18:31:36.280844 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9648bc4f-28e2-4501-b9d7-f8d04caa8550-serviceca\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-systemd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-env-overrides\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjds8\" (UniqueName: \"kubernetes.io/projected/f31a328a-30b0-4058-b64e-1d604313700e-kube-api-access-wjds8\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-ovn\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-systemd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.280959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-config\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.280993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cni-binary-copy\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-netd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-dbus\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-systemd\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-host\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-conf-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9648bc4f-28e2-4501-b9d7-f8d04caa8550-host\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-kubelet\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-system-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cnibin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281176 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-var-lib-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-bin\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-kubernetes\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-run\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaab7097-1205-4494-aa26-1c55db0a57bc-agent-certs\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-k8s-cni-cncf-io\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-netns\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-kubelet-config\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f102e-c718-4cdb-8a92-392846408a79-tmp-dir\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgcp\" (UniqueName: \"kubernetes.io/projected/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-kube-api-access-tsgcp\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cnibin\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-env-overrides\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-iptables-alerter-script\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-config\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.281608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-tmp\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-multus\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-netd\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-ovn\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9648bc4f-28e2-4501-b9d7-f8d04caa8550-host\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-kubelet\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-kubelet\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-var-lib-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-systemd-units\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-cni-bin\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovn-node-metrics-cert\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.281965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9648bc4f-28e2-4501-b9d7-f8d04caa8550-serviceca\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.282281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-systemd-units\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2hk\" (UniqueName: \"kubernetes.io/projected/aae1f017-6fe8-400b-93ed-1affbf3f56fa-kube-api-access-8w2hk\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-host-slash\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-slash\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-slash\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-etc-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6dt\" (UniqueName: \"kubernetes.io/projected/c1ee21d1-a8ea-4e75-8542-8a37effeec09-kube-api-access-2q6dt\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-host-slash\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-daemon-config\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-multus-certs\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal" event={"ID":"9f59d360c99ceabc7a2e0508d7e2b003","Type":"ContainerStarted","Data":"caf548ba52e8952659d7bbfeda6906de1b51682c56fc420fa851d2ba27b6ca6e"}
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-iptables-alerter-script\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-registration-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-registration-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-etc-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.282899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkrn\" (UniqueName: \"kubernetes.io/projected/9648bc4f-28e2-4501-b9d7-f8d04caa8550-kube-api-access-9qkrn\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-socket-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282561    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysconfig\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-conf\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-sys-fs\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-socket-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqsw\" (UniqueName: \"kubernetes.io/projected/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-kube-api-access-nwqsw\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-sys-fs\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.282942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-script-lib\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-sys\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-socket-dir-parent\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-bin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.283521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-etc-kubernetes\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-netns\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-host-run-netns\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-modprobe-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-etc-tuned\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-system-cni-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-device-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5491fda8-d309-4ca0-a8bd-4402cacb1b33-device-dir\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-node-log\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovnkube-script-lib\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-node-log\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-var-lib-kubelet\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/549f102e-c718-4cdb-8a92-392846408a79-hosts-file\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbhs\" (UniqueName: \"kubernetes.io/projected/53f382bc-f02f-4c4c-b843-abba4b52c530-kube-api-access-rbbhs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lv7\" (UniqueName: \"kubernetes.io/projected/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kube-api-access-c7lv7\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal" event={"ID":"ed46da7d6b631c06d17627b8b432d0fd","Type":"ContainerStarted","Data":"9e4cc729f3ab1309fd36a7ba240c0b1908c22a344d6f52c70db1307596ca9d70"}
Sep 09 18:31:36.284220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-log-socket\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-lib-modules\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaab7097-1205-4494-aa26-1c55db0a57bc-konnectivity-ca\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-run-openvswitch\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqv5v\" (UniqueName: \"kubernetes.io/projected/549f102e-c718-4cdb-8a92-392846408a79-kube-api-access-hqv5v\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-hostroot\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.285129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.283751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1ee21d1-a8ea-4e75-8542-8a37effeec09-log-socket\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.286818 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.286693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ee21d1-a8ea-4e75-8542-8a37effeec09-ovn-node-metrics-cert\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.290072 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.290049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6dt\" (UniqueName: \"kubernetes.io/projected/c1ee21d1-a8ea-4e75-8542-8a37effeec09-kube-api-access-2q6dt\") pod \"ovnkube-node-d4vhl\" (UID: \"c1ee21d1-a8ea-4e75-8542-8a37effeec09\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.290168 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.290156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkrn\" (UniqueName: \"kubernetes.io/projected/9648bc4f-28e2-4501-b9d7-f8d04caa8550-kube-api-access-9qkrn\") pod \"node-ca-2b2wp\" (UID: \"9648bc4f-28e2-4501-b9d7-f8d04caa8550\") " pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.290299 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.290280    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqsw\" (UniqueName: \"kubernetes.io/projected/569b6509-99ea-4f4a-8f9d-59fcb3efa8fc-kube-api-access-nwqsw\") pod \"iptables-alerter-qx427\" (UID: \"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc\") " pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.290720 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.290682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lv7\" (UniqueName: \"kubernetes.io/projected/5491fda8-d309-4ca0-a8bd-4402cacb1b33-kube-api-access-c7lv7\") pod \"aws-ebs-csi-driver-node-g4qzg\" (UID: \"5491fda8-d309-4ca0-a8bd-4402cacb1b33\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.384791 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-modprobe-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.384791 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-etc-tuned\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-system-cni-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-var-lib-kubelet\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/549f102e-c718-4cdb-8a92-392846408a79-hosts-file\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbhs\" (UniqueName: \"kubernetes.io/projected/53f382bc-f02f-4c4c-b843-abba4b52c530-kube-api-access-rbbhs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-lib-modules\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-modprobe-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaab7097-1205-4494-aa26-1c55db0a57bc-konnectivity-ca\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqv5v\" (UniqueName: \"kubernetes.io/projected/549f102e-c718-4cdb-8a92-392846408a79-kube-api-access-hqv5v\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-hostroot\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-var-lib-kubelet\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjds8\" (UniqueName: \"kubernetes.io/projected/f31a328a-30b0-4058-b64e-1d604313700e-kube-api-access-wjds8\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.384994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cni-binary-copy\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-dbus\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-systemd\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-host\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-conf-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-system-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cnibin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-kubernetes\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-run\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-system-cni-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaab7097-1205-4494-aa26-1c55db0a57bc-agent-certs\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-k8s-cni-cncf-io\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-systemd\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-k8s-cni-cncf-io\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-lib-modules\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.385526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-dbus\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-system-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-hostroot\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cnibin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-netns\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-netns\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-kubelet-config\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-host\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f102e-c718-4cdb-8a92-392846408a79-tmp-dir\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgcp\" (UniqueName: \"kubernetes.io/projected/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-kube-api-access-tsgcp\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cnibin\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-cni-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.385693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-tmp\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549f102e-c718-4cdb-8a92-392846408a79-tmp-dir\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c45724fc-0e25-4804-8f07-891fb9b57ebf-kubelet-config\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-conf-dir\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.386465 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/549f102e-c718-4cdb-8a92-392846408a79-hosts-file\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-cnibin\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-d\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-kubernetes\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-run\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-cni-binary-copy\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-multus\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-kubelet\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2hk\" (UniqueName: \"kubernetes.io/projected/aae1f017-6fe8-400b-93ed-1affbf3f56fa-kube-api-access-8w2hk\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-daemon-config\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-multus-certs\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-kubelet\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.387313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaab7097-1205-4494-aa26-1c55db0a57bc-konnectivity-ca\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysconfig\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-run-multus-certs\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-multus\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-conf\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysconfig\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-sys\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-socket-dir-parent\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-bin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-etc-kubernetes\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-sys\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae1f017-6fe8-400b-93ed-1affbf3f56fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.386973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-host-var-lib-cni-bin\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.387044    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-socket-dir-parent\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.387071    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-etc-kubernetes\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f31a328a-30b0-4058-b64e-1d604313700e-etc-sysctl-conf\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388896 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.387139    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:36.887105135 +0000 UTC m=+3.118711476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:36.388896 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.387157    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:31:36.887148081 +0000 UTC m=+3.118754423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:36.388896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-etc-tuned\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.388896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.387516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-multus-daemon-config\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.388896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.388396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaab7097-1205-4494-aa26-1c55db0a57bc-agent-certs\") pod \"konnectivity-agent-5284w\" (UID: \"aaab7097-1205-4494-aa26-1c55db0a57bc\") " pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.389183 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.389164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f31a328a-30b0-4058-b64e-1d604313700e-tmp\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.394122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.393917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgcp\" (UniqueName: \"kubernetes.io/projected/6fcf12ae-44c0-48c4-8b18-b2c314ac6749-kube-api-access-tsgcp\") pod \"multus-7xzjw\" (UID: \"6fcf12ae-44c0-48c4-8b18-b2c314ac6749\") " pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.394122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.394077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbhs\" (UniqueName: \"kubernetes.io/projected/53f382bc-f02f-4c4c-b843-abba4b52c530-kube-api-access-rbbhs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.394810 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.394775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqv5v\" (UniqueName: \"kubernetes.io/projected/549f102e-c718-4cdb-8a92-392846408a79-kube-api-access-hqv5v\") pod \"node-resolver-h868k\" (UID: \"549f102e-c718-4cdb-8a92-392846408a79\") " pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.395491 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.395420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjds8\" (UniqueName: \"kubernetes.io/projected/f31a328a-30b0-4058-b64e-1d604313700e-kube-api-access-wjds8\") pod \"tuned-6p4k8\" (UID: \"f31a328a-30b0-4058-b64e-1d604313700e\") " pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.395924 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.395904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2hk\" (UniqueName: \"kubernetes.io/projected/aae1f017-6fe8-400b-93ed-1affbf3f56fa-kube-api-access-8w2hk\") pod \"multus-additional-cni-plugins-7m9hr\" (UID: \"aae1f017-6fe8-400b-93ed-1affbf3f56fa\") " pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.397841 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.397825    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:36.397841 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.397842    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:36.397975 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.397852    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:36.397975 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.397920    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:36.897905595 +0000 UTC m=+3.129511936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:36.471263 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.471192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2b2wp"
Sep 09 18:31:36.478462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.478441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg"
Sep 09 18:31:36.483508 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.483483    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9648bc4f_28e2_4501_b9d7_f8d04caa8550.slice/crio-a04b325ea3063d48bb5de4874d3bee257fb8f050a627cbd5890c2fe61b3eb469 WatchSource:0}: Error finding container a04b325ea3063d48bb5de4874d3bee257fb8f050a627cbd5890c2fe61b3eb469: Status 404 returned error can't find the container with id a04b325ea3063d48bb5de4874d3bee257fb8f050a627cbd5890c2fe61b3eb469
Sep 09 18:31:36.485306 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.485248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qx427"
Sep 09 18:31:36.487588 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.487565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5491fda8_d309_4ca0_a8bd_4402cacb1b33.slice/crio-3e9d4b7a90e01f3755b57b4a3931989438a24049fc918f4b76f824a171fc5bfe WatchSource:0}: Error finding container 3e9d4b7a90e01f3755b57b4a3931989438a24049fc918f4b76f824a171fc5bfe: Status 404 returned error can't find the container with id 3e9d4b7a90e01f3755b57b4a3931989438a24049fc918f4b76f824a171fc5bfe
Sep 09 18:31:36.490658 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.490635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:31:36.495294 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.495269    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b6509_99ea_4f4a_8f9d_59fcb3efa8fc.slice/crio-fa205cae8892a69a3f37f5f2a4b2c01c24782f5813f49127c209c63cefac5887 WatchSource:0}: Error finding container fa205cae8892a69a3f37f5f2a4b2c01c24782f5813f49127c209c63cefac5887: Status 404 returned error can't find the container with id fa205cae8892a69a3f37f5f2a4b2c01c24782f5813f49127c209c63cefac5887
Sep 09 18:31:36.495386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.495317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:36.499694 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.499674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ee21d1_a8ea_4e75_8542_8a37effeec09.slice/crio-f2a63f0f72bde9f9317e295b975c3469b951e9e75b88ea4d9828ef74c37363cd WatchSource:0}: Error finding container f2a63f0f72bde9f9317e295b975c3469b951e9e75b88ea4d9828ef74c37363cd: Status 404 returned error can't find the container with id f2a63f0f72bde9f9317e295b975c3469b951e9e75b88ea4d9828ef74c37363cd
Sep 09 18:31:36.501079 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.501056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h868k"
Sep 09 18:31:36.504578 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.504546    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaab7097_1205_4494_aa26_1c55db0a57bc.slice/crio-9aff6b76f26fb0549a709af23a486b6e5c3158cd27a420220af249c695fadad0 WatchSource:0}: Error finding container 9aff6b76f26fb0549a709af23a486b6e5c3158cd27a420220af249c695fadad0: Status 404 returned error can't find the container with id 9aff6b76f26fb0549a709af23a486b6e5c3158cd27a420220af249c695fadad0
Sep 09 18:31:36.506837 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.506822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6p4k8"
Sep 09 18:31:36.510000 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.509982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549f102e_c718_4cdb_8a92_392846408a79.slice/crio-b517b68ddc140285cadabfcb8ca6feac17644960c7539405140dd21e6638c233 WatchSource:0}: Error finding container b517b68ddc140285cadabfcb8ca6feac17644960c7539405140dd21e6638c233: Status 404 returned error can't find the container with id b517b68ddc140285cadabfcb8ca6feac17644960c7539405140dd21e6638c233
Sep 09 18:31:36.514191 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.514176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xzjw"
Sep 09 18:31:36.514851 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.514834    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31a328a_30b0_4058_b64e_1d604313700e.slice/crio-76fa70610c6e74af62d2a67abb286175b336a25e99022a1428e35fc0d3595523 WatchSource:0}: Error finding container 76fa70610c6e74af62d2a67abb286175b336a25e99022a1428e35fc0d3595523: Status 404 returned error can't find the container with id 76fa70610c6e74af62d2a67abb286175b336a25e99022a1428e35fc0d3595523
Sep 09 18:31:36.517552 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.517534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7m9hr"
Sep 09 18:31:36.524274 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.524257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcf12ae_44c0_48c4_8b18_b2c314ac6749.slice/crio-01af1fea4b027538bb9f7195a390bab8cdde555a7b63e6e41aa0207ad5ac5556 WatchSource:0}: Error finding container 01af1fea4b027538bb9f7195a390bab8cdde555a7b63e6e41aa0207ad5ac5556: Status 404 returned error can't find the container with id 01af1fea4b027538bb9f7195a390bab8cdde555a7b63e6e41aa0207ad5ac5556
Sep 09 18:31:36.526474 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:31:36.526449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae1f017_6fe8_400b_93ed_1affbf3f56fa.slice/crio-d27345803332e9cfaae845168389c10947f6caa22021b13e701c9bf88aab82dd WatchSource:0}: Error finding container d27345803332e9cfaae845168389c10947f6caa22021b13e701c9bf88aab82dd: Status 404 returned error can't find the container with id d27345803332e9cfaae845168389c10947f6caa22021b13e701c9bf88aab82dd
Sep 09 18:31:36.612474 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.612442    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:36.889527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.889452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:36.889527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.889489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:36.889700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.889599    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:36.889700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.889601    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:36.889700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.889661    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:31:37.889645496 +0000 UTC m=+4.121251847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:36.889700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.889675    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:37.889669989 +0000 UTC m=+4.121276331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:36.990578 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:36.990546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:36.990771 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.990720    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:36.990771 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.990745    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:36.990771 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.990757    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:36.990930 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:36.990818    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:37.990799936 +0000 UTC m=+4.222406279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:37.212402 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.212306    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 18:26:36 +0000 UTC" deadline="2028-06-08 10:40:36.002843128 +0000 UTC"
Sep 09 18:31:37.212402 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.212355    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15304h8m58.790492887s"
Sep 09 18:31:37.278083 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.278047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:37.278248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:37.278185    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:37.287257 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.287228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg" event={"ID":"5491fda8-d309-4ca0-a8bd-4402cacb1b33","Type":"ContainerStarted","Data":"3e9d4b7a90e01f3755b57b4a3931989438a24049fc918f4b76f824a171fc5bfe"}
Sep 09 18:31:37.289014 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.288991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6p4k8" event={"ID":"f31a328a-30b0-4058-b64e-1d604313700e","Type":"ContainerStarted","Data":"76fa70610c6e74af62d2a67abb286175b336a25e99022a1428e35fc0d3595523"}
Sep 09 18:31:37.290608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.290585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h868k" event={"ID":"549f102e-c718-4cdb-8a92-392846408a79","Type":"ContainerStarted","Data":"b517b68ddc140285cadabfcb8ca6feac17644960c7539405140dd21e6638c233"}
Sep 09 18:31:37.292833 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.292811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"f2a63f0f72bde9f9317e295b975c3469b951e9e75b88ea4d9828ef74c37363cd"}
Sep 09 18:31:37.294605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.294582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qx427" event={"ID":"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc","Type":"ContainerStarted","Data":"fa205cae8892a69a3f37f5f2a4b2c01c24782f5813f49127c209c63cefac5887"}
Sep 09 18:31:37.296271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.296247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2b2wp" event={"ID":"9648bc4f-28e2-4501-b9d7-f8d04caa8550","Type":"ContainerStarted","Data":"a04b325ea3063d48bb5de4874d3bee257fb8f050a627cbd5890c2fe61b3eb469"}
Sep 09 18:31:37.298386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.298361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerStarted","Data":"d27345803332e9cfaae845168389c10947f6caa22021b13e701c9bf88aab82dd"}
Sep 09 18:31:37.300032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.300002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xzjw" event={"ID":"6fcf12ae-44c0-48c4-8b18-b2c314ac6749","Type":"ContainerStarted","Data":"01af1fea4b027538bb9f7195a390bab8cdde555a7b63e6e41aa0207ad5ac5556"}
Sep 09 18:31:37.302522 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.302496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5284w" event={"ID":"aaab7097-1205-4494-aa26-1c55db0a57bc","Type":"ContainerStarted","Data":"9aff6b76f26fb0549a709af23a486b6e5c3158cd27a420220af249c695fadad0"}
Sep 09 18:31:37.899027 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.898987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:37.899197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:37.899055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:37.899260 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:37.899195    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:37.899260 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:37.899254    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:31:39.899236984 +0000 UTC m=+6.130843330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:37.899659 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:37.899637    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:37.899767 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:37.899694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:39.899679026 +0000 UTC m=+6.131285374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:38.000687 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.000334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:38.000687 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.000557    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:38.000687 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.000579    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:38.000687 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.000592    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:38.000687 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.000652    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:40.000633762 +0000 UTC m=+6.232240119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:38.126446 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.126415    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:38.214352 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.212663    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 18:26:36 +0000 UTC" deadline="2028-06-19 05:40:16.380995428 +0000 UTC"
Sep 09 18:31:38.214352 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.212735    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15563h8m38.168265505s"
Sep 09 18:31:38.280669 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.280594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:38.280834 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.280751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:38.281137 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:38.281119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:38.281224 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:38.281206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:39.278348 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:39.277889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:39.278348 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:39.278012    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:39.315668 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:39.314972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal" event={"ID":"ed46da7d6b631c06d17627b8b432d0fd","Type":"ContainerStarted","Data":"1a9c56f1601c197074926fd3ae972cd9bc4d15009981d43deb8a6fb7491b75ce"}
Sep 09 18:31:39.919156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:39.919124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:39.919325 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:39.919173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:39.919325 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:39.919312    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:39.919442 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:39.919369    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:31:43.919350366 +0000 UTC m=+10.150956726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:39.919883 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:39.919781    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:39.919883 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:39.919835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:43.919819754 +0000 UTC m=+10.151426101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:40.019950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:40.019918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:40.020125 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.020048    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:40.020125 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.020068    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:40.020125 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.020077    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:40.020278 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.020127    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:44.020111534 +0000 UTC m=+10.251717894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:40.279430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:40.278833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:40.279430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:40.278842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:40.279430 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.279003    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:40.279430 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:40.279104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:40.321803 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:40.321757    2581 generic.go:356] "Generic (PLEG): container finished" podID="ed46da7d6b631c06d17627b8b432d0fd" containerID="1a9c56f1601c197074926fd3ae972cd9bc4d15009981d43deb8a6fb7491b75ce" exitCode=0
Sep 09 18:31:40.322592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:40.322550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal" event={"ID":"ed46da7d6b631c06d17627b8b432d0fd","Type":"ContainerDied","Data":"1a9c56f1601c197074926fd3ae972cd9bc4d15009981d43deb8a6fb7491b75ce"}
Sep 09 18:31:41.278346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:41.278310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:41.278547 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:41.278447    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:42.279867 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:42.279833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:42.280254 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:42.279959    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:42.280254 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:42.280042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:42.280254 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:42.280120    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:43.277950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:43.277917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:43.278114 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:43.278051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:43.952763 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:43.952724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:43.953210 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:43.952777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:43.953210 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:43.952832    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:43.953210 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:43.952907    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:51.952886509 +0000 UTC m=+18.184492851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:43.953210 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:43.952926    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:43.953210 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:43.952976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:31:51.952960265 +0000 UTC m=+18.184566610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:44.053374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:44.053333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:44.053568 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.053539    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:44.053700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.053604    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:44.053700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.053619    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:44.053700 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.053683    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:52.053665941 +0000 UTC m=+18.285272281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:44.280675 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:44.280600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:44.280836 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.280742    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:44.280836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:44.280811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:44.280934 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:44.280904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:45.278344 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:45.278308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:45.278765 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:45.278422    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:46.278638 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:46.278605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:46.279071 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:46.278746    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:46.279071 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:46.278923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:46.279071 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:46.279021    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:47.278472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:47.278437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:47.278668 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:47.278564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:48.277718 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:48.277672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:48.277926 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:48.277697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:48.277926 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:48.277807    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:48.278034 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:48.277960    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:49.277636 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:49.277600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:49.278096 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:49.277718    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:50.278131 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:50.278097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:50.278475 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:50.278202    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:50.278475 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:50.278260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:50.278475 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:50.278361    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:51.278247 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:51.278205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:51.278690 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:51.278341    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:52.009085 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:52.009047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:52.009276 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:52.009098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:52.009276 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.009216    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:52.009276 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.009218    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:52.009415 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.009285    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:08.009269275 +0000 UTC m=+34.240875615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:52.009415 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.009301    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:32:08.009292409 +0000 UTC m=+34.240898750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:52.109919 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:52.109817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:52.110088 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.110008    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:52.110088 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.110036    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:52.110088 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.110051    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:52.110302 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.110104    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:08.11008932 +0000 UTC m=+34.341695661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:52.278427 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:52.278343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:52.278848 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.278487    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:52.278988 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:52.278970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:52.279090 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:52.279071    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:53.277664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:53.277626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:53.277838 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:53.277771    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:54.278986 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:54.278960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:54.279397 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:54.279047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:54.279453 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:54.279424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:54.279518 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:54.279508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:55.278064 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:55.278005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:55.278177 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:55.278099    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:56.279124 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.278643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:56.279824 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.278690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:56.279824 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:56.279193    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:56.279824 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:56.279235    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:56.355813 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.355774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal" event={"ID":"9f59d360c99ceabc7a2e0508d7e2b003","Type":"ContainerStarted","Data":"63a40ca7794c19d09e3978c7ecba9f0f4d652b244d78b170c763bb57ffd0da9b"}
Sep 09 18:31:56.358225 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.358196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal" event={"ID":"ed46da7d6b631c06d17627b8b432d0fd","Type":"ContainerStarted","Data":"c743f1c86b608237dea504cff00757b055abda896fae1f40a0735dd9fba07d58"}
Sep 09 18:31:56.360025 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.360002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6p4k8" event={"ID":"f31a328a-30b0-4058-b64e-1d604313700e","Type":"ContainerStarted","Data":"24f45cdc7fef24a0af12d978800493775988a7315d8ba326c4929dd4d21a992c"}
Sep 09 18:31:56.361615 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.361585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h868k" event={"ID":"549f102e-c718-4cdb-8a92-392846408a79","Type":"ContainerStarted","Data":"b028b8b5b7a08aa9ab3f79d2f9e29f8d5a919d6b79894b00a95cae726658b249"}
Sep 09 18:31:56.365056 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"4c74059a9595706f64a88d45c79b943ce76c3f9be9aa9b22316098b98ace78a8"}
Sep 09 18:31:56.365153 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"e14305707d0d995a1d4a755ed08b3b714f62281bfeb2709a6c1061606eea7429"}
Sep 09 18:31:56.365153 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"c585383268b3f4865c6b11cd09a8758ec6381c34d1951ea6291327f84ec95a75"}
Sep 09 18:31:56.365271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"21f084db7c811a9220d00e1affa332ac0ac349369f95da8a98a8e2d4b47527f7"}
Sep 09 18:31:56.365271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"448046fd6e984b8a19788c238ed51a77688761deccee1ad827e7d20d246fef67"}
Sep 09 18:31:56.365271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.365267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"3704a974a7117926d7bf1242cb0f5a856b90d966e6eeeb8ae82f7538418f4c8c"}
Sep 09 18:31:56.366662 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.366640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2b2wp" event={"ID":"9648bc4f-28e2-4501-b9d7-f8d04caa8550","Type":"ContainerStarted","Data":"f5b7a50dea5a343b76d0ec2841f48921e4716e08314ab2dd1b3834f7905c5cde"}
Sep 09 18:31:56.368503 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.368481    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="97bcf6a4abcfb8ae95dee437b756f73092eb237479160e2f67125e496c7afa9d" exitCode=0
Sep 09 18:31:56.368592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.368566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"97bcf6a4abcfb8ae95dee437b756f73092eb237479160e2f67125e496c7afa9d"}
Sep 09 18:31:56.370666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.370640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xzjw" event={"ID":"6fcf12ae-44c0-48c4-8b18-b2c314ac6749","Type":"ContainerStarted","Data":"55b6411cac3add62d5ac0014f2a791382e2a115a029a52791c352952ae9c878e"}
Sep 09 18:31:56.372334 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.372310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5284w" event={"ID":"aaab7097-1205-4494-aa26-1c55db0a57bc","Type":"ContainerStarted","Data":"37f33c56d40982e00919e6dcda691d9e862879d27a4c07bbe0a34d30a9c29079"}
Sep 09 18:31:56.373917 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.373899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg" event={"ID":"5491fda8-d309-4ca0-a8bd-4402cacb1b33","Type":"ContainerStarted","Data":"da4640df7f3a092e70c4a7735a1757704b1987198957490865f349f63bc9a715"}
Sep 09 18:31:56.385633 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.385570    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-104.ec2.internal" podStartSLOduration=21.385550005 podStartE2EDuration="21.385550005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.370959999 +0000 UTC m=+22.602566378" watchObservedRunningTime="2026-09-09 18:31:56.385550005 +0000 UTC m=+22.617156348"
Sep 09 18:31:56.386172 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.386133    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-104.ec2.internal" podStartSLOduration=21.386125467 podStartE2EDuration="21.386125467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.385834008 +0000 UTC m=+22.617440387" watchObservedRunningTime="2026-09-09 18:31:56.386125467 +0000 UTC m=+22.617731827"
Sep 09 18:31:56.401776 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.401701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h868k" podStartSLOduration=8.607513174 podStartE2EDuration="22.401693242s" totalImagesPullingTime="13.794180068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.40139253 +0000 UTC m=+22.632998893" watchObservedRunningTime="2026-09-09 18:31:56.401693242 +0000 UTC m=+22.633299603"
Sep 09 18:31:56.430839 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.430796    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2b2wp" podStartSLOduration=3.916606015 podStartE2EDuration="22.430782803s" totalImagesPullingTime="18.514176788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.430645768 +0000 UTC m=+22.662252131" watchObservedRunningTime="2026-09-09 18:31:56.430782803 +0000 UTC m=+22.662389156"
Sep 09 18:31:56.445981 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.445937    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6p4k8" podStartSLOduration=3.6628283699999997 podStartE2EDuration="22.445922335s" totalImagesPullingTime="18.783093965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.445353894 +0000 UTC m=+22.676960257" watchObservedRunningTime="2026-09-09 18:31:56.445922335 +0000 UTC m=+22.677528698"
Sep 09 18:31:56.480979 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.480942    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5284w" podStartSLOduration=3.67993852 podStartE2EDuration="22.480931034s" totalImagesPullingTime="18.800992514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.480820435 +0000 UTC m=+22.712426808" watchObservedRunningTime="2026-09-09 18:31:56.480931034 +0000 UTC m=+22.712537396"
Sep 09 18:31:56.481125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:56.481108    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7xzjw" podStartSLOduration=3.706062721 podStartE2EDuration="22.481104516s" totalImagesPullingTime="18.775041795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.461020534 +0000 UTC m=+22.692626899" watchObservedRunningTime="2026-09-09 18:31:56.481104516 +0000 UTC m=+22.712710877"
Sep 09 18:31:57.085489 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.085446    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:57.086021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.086006    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:57.233373 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.232786    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 18:31:57.257308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.257214    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T18:31:57.232984Z","UUID":"a8d25b4d-a95f-4e4f-aaa8-d9fde561b52e","Handler":null,"Name":"","Endpoint":""}
Sep 09 18:31:57.258860 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.258839    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 18:31:57.258992 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.258877    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 18:31:57.278079 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.278057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:57.278180 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:57.278144    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:57.378526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.378440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg" event={"ID":"5491fda8-d309-4ca0-a8bd-4402cacb1b33","Type":"ContainerStarted","Data":"f030d6e129237a35984dd4288a52855c81edbd76e76eadc97ebc59aea09f8ff7"}
Sep 09 18:31:57.380140 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.380113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qx427" event={"ID":"569b6509-99ea-4f4a-8f9d-59fcb3efa8fc","Type":"ContainerStarted","Data":"0af4648f017a9edba900d32c89c6c9502a815c9eaa3062016ac25b47b93d7cf6"}
Sep 09 18:31:57.394424 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:57.394371    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qx427" podStartSLOduration=4.592038852 podStartE2EDuration="23.394352474s" totalImagesPullingTime="18.802313622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:57.394163234 +0000 UTC m=+23.625769597" watchObservedRunningTime="2026-09-09 18:31:57.394352474 +0000 UTC m=+23.625958833"
Sep 09 18:31:58.278823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.278747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:31:58.278823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.278797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:31:58.279039 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:58.278882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:31:58.279039 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:58.278955    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:31:58.386729 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.386686    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 18:31:58.387099 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.386728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"f9279a1deef0ab44ff95929aeda9f0e61dce944f460d34612c2d2084cb33fa7d"}
Sep 09 18:31:58.664948 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.664865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:58.665468 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:58.665423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5284w"
Sep 09 18:31:59.278723 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:59.278678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:31:59.278899 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:31:59.278801    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:31:59.391450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:59.391416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg" event={"ID":"5491fda8-d309-4ca0-a8bd-4402cacb1b33","Type":"ContainerStarted","Data":"5458a766a1968929295a6647597a42fdc8ea9c81937809314be0348cebb86d09"}
Sep 09 18:31:59.407537 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:31:59.407479    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4qzg" podStartSLOduration=4.163902122 podStartE2EDuration="25.40746321s" totalImagesPullingTime="21.243561088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:59.40742872 +0000 UTC m=+25.639035082" watchObservedRunningTime="2026-09-09 18:31:59.40746321 +0000 UTC m=+25.639069574"
Sep 09 18:32:00.278136 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:00.278105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:00.278308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:00.278105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:00.278308 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:00.278255    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:32:00.278308 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:00.278282    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:32:00.399462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:00.399378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" event={"ID":"c1ee21d1-a8ea-4e75-8542-8a37effeec09","Type":"ContainerStarted","Data":"6ae24ec3d3c80ab43055d25a4191ec42c9a874e6ef5b1ab6026195a495948e6b"}
Sep 09 18:32:00.424744 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:00.424668    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl" podStartSLOduration=7.563880081 podStartE2EDuration="26.424647424s" totalImagesPullingTime="18.860767343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:00.42375282 +0000 UTC m=+26.655359182" watchObservedRunningTime="2026-09-09 18:32:00.424647424 +0000 UTC m=+26.656253786"
Sep 09 18:32:01.278315 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.278138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:01.278482 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:01.278409    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:32:01.402594 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.402572    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 18:32:01.403187 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.403116    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:01.403271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.403192    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:01.422929 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.422390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:01.424585 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:01.424560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:02.082109 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.082080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hhwv"]
Sep 09 18:32:02.082290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.082223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:02.082352 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:02.082313    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:32:02.084295 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.084264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkdph"]
Sep 09 18:32:02.084433 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.084381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:02.084508 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:02.084494    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:32:02.084971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.084947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tp4fk"]
Sep 09 18:32:02.085096 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.085044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:02.085157 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:02.085137    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:32:02.404794 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:02.404701    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 18:32:03.278591 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:03.278563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:03.278774 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:03.278663    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:32:03.408925 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:03.408892    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="60205146b00e123bee4749bdf5f1bea90f7a4b0898e44c9a4e81a41438201ff1" exitCode=0
Sep 09 18:32:03.409408 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:03.408961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"60205146b00e123bee4749bdf5f1bea90f7a4b0898e44c9a4e81a41438201ff1"}
Sep 09 18:32:03.409408 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:03.409018    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 18:32:04.278962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:04.278929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:04.279105 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:04.278974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:04.279105 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:04.279061    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:32:04.279194 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:04.279157    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:32:05.278065 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:05.277989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:05.278504 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:05.278088    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:32:06.203496 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:06.203284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:06.203681 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:06.203668    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 18:32:06.222903 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:06.222882    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4vhl"
Sep 09 18:32:06.278342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:06.278315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:06.278673 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:06.278431    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:32:06.278673 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:06.278503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:06.278673 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:06.278589    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp4fk" podUID="ead1af0a-36bf-4fa6-9020-a73b9f4a8312"
Sep 09 18:32:07.278433 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:07.278394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:07.278850 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:07.278503    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hhwv" podUID="c45724fc-0e25-4804-8f07-891fb9b57ebf"
Sep 09 18:32:07.418869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:07.418779    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="d75b95ecb7918e2a56d3687ee655b1bf51bb3b63246c000ccba2d819e5401abb" exitCode=0
Sep 09 18:32:07.418869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:07.418826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"d75b95ecb7918e2a56d3687ee655b1bf51bb3b63246c000ccba2d819e5401abb"}
Sep 09 18:32:08.024776 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.024742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:08.024776 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.024780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:08.024978 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.024881    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:08.024978 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.024888    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:08.024978 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.024944    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret podName:c45724fc-0e25-4804-8f07-891fb9b57ebf nodeName:}" failed. No retries permitted until 2026-09-09 18:32:40.024919335 +0000 UTC m=+66.256525674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret") pod "global-pull-secret-syncer-9hhwv" (UID: "c45724fc-0e25-4804-8f07-891fb9b57ebf") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:08.024978 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.024962    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:40.024952799 +0000 UTC m=+66.256559139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:08.088154 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.088073    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-104.ec2.internal" event="NodeReady"
Sep 09 18:32:08.088290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.088201    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 18:32:08.120470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.120441    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:32:08.122262 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.122248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.124698 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.124660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-74w6f\""
Sep 09 18:32:08.124951 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.124912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 18:32:08.125022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.124975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 18:32:08.125022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.125001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:08.125161 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.125069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 18:32:08.125161 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.125145    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:32:08.125275 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.125163    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:32:08.125275 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.125177    2581 projected.go:196] Error preparing data for projected volume kube-api-access-49s5c for pod openshift-network-diagnostics/network-check-target-tp4fk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:08.125275 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.125223    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c podName:ead1af0a-36bf-4fa6-9020-a73b9f4a8312 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:40.125209393 +0000 UTC m=+66.356815746 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-49s5c" (UniqueName: "kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c") pod "network-check-target-tp4fk" (UID: "ead1af0a-36bf-4fa6-9020-a73b9f4a8312") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:08.130615 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.130594    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 18:32:08.130778 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.130761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qm8n4"]
Sep 09 18:32:08.132409 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.132396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.133563 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.133542    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vf9rh"]
Sep 09 18:32:08.134512 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.134498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 18:32:08.134894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.134878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 18:32:08.134980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.134912    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 18:32:08.134980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.134921    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4r989\""
Sep 09 18:32:08.135650 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.135615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.137011 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.136992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:32:08.137877 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.137863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 18:32:08.138021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.138005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 18:32:08.138115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.138100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgjhb\""
Sep 09 18:32:08.151999 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.151978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vf9rh"]
Sep 09 18:32:08.152430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.152414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qm8n4"]
Sep 09 18:32:08.225425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.225568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqst\" (UniqueName: \"kubernetes.io/projected/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-kube-api-access-rhqst\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.225568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-cert\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5hc\" (UniqueName: \"kubernetes.io/projected/0f3173bc-3ab6-4b3b-889f-8c95808239f7-kube-api-access-ns5hc\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828fz\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3173bc-3ab6-4b3b-889f-8c95808239f7-config-volume\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.225788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.225749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3173bc-3ab6-4b3b-889f-8c95808239f7-tmp-dir\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.277956 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.277932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:08.278035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.277932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:08.280572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.280551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 18:32:08.280962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.280587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 18:32:08.280962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.280610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2784s\""
Sep 09 18:32:08.280962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.280611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 18:32:08.280962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.280653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdbl9\""
Sep 09 18:32:08.326198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqst\" (UniqueName: \"kubernetes.io/projected/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-kube-api-access-rhqst\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-cert\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.326603 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.326336    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:08.326603 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.326403    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:08.826382387 +0000 UTC m=+35.057988742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:08.326603 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326603 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326603 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5hc\" (UniqueName: \"kubernetes.io/projected/0f3173bc-3ab6-4b3b-889f-8c95808239f7-kube-api-access-ns5hc\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828fz\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.326651    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.326683    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3173bc-3ab6-4b3b-889f-8c95808239f7-config-volume\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.326750    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:08.826735656 +0000 UTC m=+35.058342011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3173bc-3ab6-4b3b-889f-8c95808239f7-tmp-dir\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.326879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.326848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.327328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.327243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3173bc-3ab6-4b3b-889f-8c95808239f7-tmp-dir\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.327328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.327272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f3173bc-3ab6-4b3b-889f-8c95808239f7-config-volume\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.327423 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.327384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.327765 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.327747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.330623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.330602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.330623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.330610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.330798 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.330722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-cert\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.335066 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.335033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.335441 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.335404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqst\" (UniqueName: \"kubernetes.io/projected/80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3-kube-api-access-rhqst\") pod \"ingress-canary-qm8n4\" (UID: \"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3\") " pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.335693 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.335660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5hc\" (UniqueName: \"kubernetes.io/projected/0f3173bc-3ab6-4b3b-889f-8c95808239f7-kube-api-access-ns5hc\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.335812 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.335775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828fz\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.442780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.442686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qm8n4"
Sep 09 18:32:08.716811 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.716737    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qm8n4"]
Sep 09 18:32:08.721044 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:08.721020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80737e9b_b5fe_4d8e_a8e3_d8ee4984fee3.slice/crio-15d2a32bf9e90e105c80e844bf3ab4942beba9229a68be7e906089e90affcb6c WatchSource:0}: Error finding container 15d2a32bf9e90e105c80e844bf3ab4942beba9229a68be7e906089e90affcb6c: Status 404 returned error can't find the container with id 15d2a32bf9e90e105c80e844bf3ab4942beba9229a68be7e906089e90affcb6c
Sep 09 18:32:08.830509 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.830351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:08.830509 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.830494    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:08.830509 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.830514    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:08.830828 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:08.830561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:08.830828 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.830605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:09.830585651 +0000 UTC m=+36.062191993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:08.830828 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.830650    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:08.830828 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:08.830735    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:09.830695868 +0000 UTC m=+36.062302237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:09.101637 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.101554    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:09.184886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.184845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:09.277841 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.277816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:09.280426 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.280405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 18:32:09.424959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.424876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qm8n4" event={"ID":"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3","Type":"ContainerStarted","Data":"15d2a32bf9e90e105c80e844bf3ab4942beba9229a68be7e906089e90affcb6c"}
Sep 09 18:32:09.838270 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.838181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:09.838270 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.838242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:09.838494 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:09.838352    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:09.838494 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:09.838431    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:11.838411942 +0000 UTC m=+38.070018306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:09.838494 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:09.838432    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:09.838494 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:09.838452    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:09.838682 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:09.838550    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:11.838533342 +0000 UTC m=+38.070139695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:09.884974 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.884940    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:09.985684 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:09.985654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:10.578666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:10.578629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:10.632116 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:10.632073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:10.691674 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:10.691631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:11.041521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.041491    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"]
Sep 09 18:32:11.044571 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.044531    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"]
Sep 09 18:32:11.047332 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.045569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.048619    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"]
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.048813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.048910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.049319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.049425    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.050146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l8r2b\""
Sep 09 18:32:11.051044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.050361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 18:32:11.052359 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.052329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 18:32:11.053911 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.053888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"]
Sep 09 18:32:11.054047 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.053959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.056928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.056267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 18:32:11.056928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.056383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 18:32:11.056928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.056387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 18:32:11.056928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.056613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 18:32:11.059766 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.059747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"]
Sep 09 18:32:11.067863 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.067448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"]
Sep 09 18:32:11.149632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f7b4890-b6fb-4fe4-8705-225643b2f36e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.149806 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.149806 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98kq\" (UniqueName: \"kubernetes.io/projected/1f406003-51a2-4e47-84ae-0debd381f555-kube-api-access-w98kq\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.149806 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.149916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9qg\" (UniqueName: \"kubernetes.io/projected/9f7b4890-b6fb-4fe4-8705-225643b2f36e-kube-api-access-pc9qg\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.149916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f406003-51a2-4e47-84ae-0debd381f555-additional-service-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.149916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfs8\" (UniqueName: \"kubernetes.io/projected/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-kube-api-access-pzfs8\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.150004 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.150004 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.150004 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.149998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-tmp\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.150090 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.150016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.251242 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.251371 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f7b4890-b6fb-4fe4-8705-225643b2f36e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.251371 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.251489 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98kq\" (UniqueName: \"kubernetes.io/projected/1f406003-51a2-4e47-84ae-0debd381f555-kube-api-access-w98kq\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.251548 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.251600 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9qg\" (UniqueName: \"kubernetes.io/projected/9f7b4890-b6fb-4fe4-8705-225643b2f36e-kube-api-access-pc9qg\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f406003-51a2-4e47-84ae-0debd381f555-additional-service-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfs8\" (UniqueName: \"kubernetes.io/projected/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-kube-api-access-pzfs8\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.251914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-tmp\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.252584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.252268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-tmp\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.253018 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.252545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f406003-51a2-4e47-84ae-0debd381f555-additional-service-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.255247 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-ca\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.255386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.255429 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.255463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.255497 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f7b4890-b6fb-4fe4-8705-225643b2f36e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.255527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.255484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1f406003-51a2-4e47-84ae-0debd381f555-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.260917 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.260887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9qg\" (UniqueName: \"kubernetes.io/projected/9f7b4890-b6fb-4fe4-8705-225643b2f36e-kube-api-access-pc9qg\") pod \"managed-serviceaccount-addon-agent-658778bc76-7phwf\" (UID: \"9f7b4890-b6fb-4fe4-8705-225643b2f36e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.261008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.260984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfs8\" (UniqueName: \"kubernetes.io/projected/d8da06e8-50eb-41b1-9b5e-c7f722ad7831-kube-api-access-pzfs8\") pod \"klusterlet-addon-workmgr-dc5676f5b-xvt94\" (UID: \"d8da06e8-50eb-41b1-9b5e-c7f722ad7831\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.261258 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.261239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98kq\" (UniqueName: \"kubernetes.io/projected/1f406003-51a2-4e47-84ae-0debd381f555-kube-api-access-w98kq\") pod \"cluster-proxy-proxy-agent-574b4cdf6f-w8l98\" (UID: \"1f406003-51a2-4e47-84ae-0debd381f555\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.368081 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.368042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:32:11.376894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.376870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:11.383955 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.383916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:32:11.437596 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.437489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qm8n4" event={"ID":"80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3","Type":"ContainerStarted","Data":"37b353a133241ef9d2688433f06211f37e70e698a5a20e05011c30ed9785908e"}
Sep 09 18:32:11.654850 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.654790    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qm8n4" podStartSLOduration=1.128712814 podStartE2EDuration="3.654764855s" totalImagesPullingTime="2.526052041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:11.455565442 +0000 UTC m=+37.687171805" watchObservedRunningTime="2026-09-09 18:32:11.654764855 +0000 UTC m=+37.886371243"
Sep 09 18:32:11.655439 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.655290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"]
Sep 09 18:32:11.658766 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:11.658741    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7b4890_b6fb_4fe4_8705_225643b2f36e.slice/crio-9dc356e1e7f35d7098e5f8242e157a93ffcfbc0deab1c4969904631c856ee8a1 WatchSource:0}: Error finding container 9dc356e1e7f35d7098e5f8242e157a93ffcfbc0deab1c4969904631c856ee8a1: Status 404 returned error can't find the container with id 9dc356e1e7f35d7098e5f8242e157a93ffcfbc0deab1c4969904631c856ee8a1
Sep 09 18:32:11.670677 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.670654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"]
Sep 09 18:32:11.673549 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:11.673526    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da06e8_50eb_41b1_9b5e_c7f722ad7831.slice/crio-9aafa49be67b73e46cd6614fd74664a930634a44fa9a60f26117acd17321f1cf WatchSource:0}: Error finding container 9aafa49be67b73e46cd6614fd74664a930634a44fa9a60f26117acd17321f1cf: Status 404 returned error can't find the container with id 9aafa49be67b73e46cd6614fd74664a930634a44fa9a60f26117acd17321f1cf
Sep 09 18:32:11.679407 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.679385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"]
Sep 09 18:32:11.682617 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:11.682594    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f406003_51a2_4e47_84ae_0debd381f555.slice/crio-9d9dc1a6c8bdca56d419fc14c05498f86bebe371ae5c8ceceb74c2c041c213c8 WatchSource:0}: Error finding container 9d9dc1a6c8bdca56d419fc14c05498f86bebe371ae5c8ceceb74c2c041c213c8: Status 404 returned error can't find the container with id 9d9dc1a6c8bdca56d419fc14c05498f86bebe371ae5c8ceceb74c2c041c213c8
Sep 09 18:32:11.857764 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.857732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:11.857950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.857776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:11.857950 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:11.857887    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:11.857950 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:11.857926    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:11.857950 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:11.857940    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:11.857950 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:11.857950    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:15.857935209 +0000 UTC m=+42.089541549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:11.858205 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:11.857980    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:15.857969966 +0000 UTC m=+42.089576306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:11.873026 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:11.873001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:12.445128 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:12.444801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerStarted","Data":"9aafa49be67b73e46cd6614fd74664a930634a44fa9a60f26117acd17321f1cf"}
Sep 09 18:32:12.448436 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:12.448410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerStarted","Data":"9d9dc1a6c8bdca56d419fc14c05498f86bebe371ae5c8ceceb74c2c041c213c8"}
Sep 09 18:32:12.450414 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:12.450369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerStarted","Data":"9dc356e1e7f35d7098e5f8242e157a93ffcfbc0deab1c4969904631c856ee8a1"}
Sep 09 18:32:13.630747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:13.630701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:13.678581 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:13.678555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:15.905182 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:15.905139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:15.905202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:15.905324    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:15.905350    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:15.905369    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:15.905404    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:23.905380926 +0000 UTC m=+50.136987280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:15.905657 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:15.905425    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:23.905415631 +0000 UTC m=+50.137021975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:18.472687 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.472648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerStarted","Data":"9c82d0d779a696117126264342f47236ffd787ef0588217592403a2bb8ed6a37"}
Sep 09 18:32:18.476188 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.476125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerStarted","Data":"a76308c4ea7665d63ccbeb2b85f65ff2c2082c449026ff923f23513d341e01b4"}
Sep 09 18:32:18.477916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.477891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerStarted","Data":"190eb2290b78b24912239d7ebbafcc4e005fb5de4079bd5d3c0fb97ea6275045"}
Sep 09 18:32:18.478137 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.478091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:18.480022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.480000    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:32:18.488168 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.488126    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" podStartSLOduration=1.444583967 podStartE2EDuration="7.488109357s" totalImagesPullingTime="6.04352539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:18.486545626 +0000 UTC m=+44.718151994" watchObservedRunningTime="2026-09-09 18:32:18.488109357 +0000 UTC m=+44.719715721"
Sep 09 18:32:18.501496 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:18.501455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" podStartSLOduration=1.472463409 podStartE2EDuration="7.501442289s" totalImagesPullingTime="6.02897888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:18.500832228 +0000 UTC m=+44.732438591" watchObservedRunningTime="2026-09-09 18:32:18.501442289 +0000 UTC m=+44.733048706"
Sep 09 18:32:19.482520 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:19.482445    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="a76308c4ea7665d63ccbeb2b85f65ff2c2082c449026ff923f23513d341e01b4" exitCode=0
Sep 09 18:32:19.482983 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:19.482533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"a76308c4ea7665d63ccbeb2b85f65ff2c2082c449026ff923f23513d341e01b4"}
Sep 09 18:32:20.486772 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:20.486665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerStarted","Data":"a41a46992159c07449a8e40d71ffe7a4966cdb19a6f697a40c73019ebd488ba7"}
Sep 09 18:32:20.486772 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:20.486728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerStarted","Data":"bafc60e774fda82d59d29fe1c2bf85e6aaaaff1853844c9c505898ff91f66d64"}
Sep 09 18:32:20.486772 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:20.486747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerStarted","Data":"2920d236f93c99ac180ee4f3a2e74d7c5f659b7a31fdf5d0e6ecbce05829a13b"}
Sep 09 18:32:20.504488 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:20.504437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" podStartSLOduration=1.657897512 podStartE2EDuration="9.504423615s" totalImagesPullingTime="7.846526103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:20.503155687 +0000 UTC m=+46.734762048" watchObservedRunningTime="2026-09-09 18:32:20.504423615 +0000 UTC m=+46.736029977"
Sep 09 18:32:21.530472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:21.530442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:21.585274 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:21.585245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:23.607421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:23.607386    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:23.656822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:23.656787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:23.972050 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:23.971959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:23.972050 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:23.972013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:23.972248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:23.972133    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:23.972248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:23.972188    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:23.972248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:23.972203    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:23.972248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:23.972215    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:39.972192899 +0000 UTC m=+66.203799253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:23.972248 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:23.972241    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:39.972229343 +0000 UTC m=+66.203835699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:25.501836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:25.501801    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="667aa6aba6f82bf73058e0a27cb39abcf370f11802a1d58fa03dede9403f73fd" exitCode=0
Sep 09 18:32:25.502223 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:25.501861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"667aa6aba6f82bf73058e0a27cb39abcf370f11802a1d58fa03dede9403f73fd"}
Sep 09 18:32:26.507542 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:26.507504    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae1f017-6fe8-400b-93ed-1affbf3f56fa" containerID="9b3fc8d43071b57a169a27072bb3bc8f77d44aed89432f6ffa4339ba6cd2e844" exitCode=0
Sep 09 18:32:26.507985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:26.507591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerDied","Data":"9b3fc8d43071b57a169a27072bb3bc8f77d44aed89432f6ffa4339ba6cd2e844"}
Sep 09 18:32:26.988493 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:26.988457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:27.046412 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:27.046387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:27.514443 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:27.514410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" event={"ID":"aae1f017-6fe8-400b-93ed-1affbf3f56fa","Type":"ContainerStarted","Data":"c5dfbb346f48151f784f4cd61cef7ba664ebf3be753a379bf00186a033e3554d"}
Sep 09 18:32:27.536983 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:27.536940    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7m9hr" podStartSLOduration=9.835234197 podStartE2EDuration="53.536925807s" totalImagesPullingTime="43.70169161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:27.535345318 +0000 UTC m=+53.766951679" watchObservedRunningTime="2026-09-09 18:32:27.536925807 +0000 UTC m=+53.768532168"
Sep 09 18:32:27.655622 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:27.655591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:27.755250 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:27.755215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:28.655183 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:28.655156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:28.708070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:28.708038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:37.341080 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:37.341033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:37.391066 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:37.391034    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:39.982255 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:39.982196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:32:39.982255 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:39.982266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:32:39.982790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:39.982401    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:39.982790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:39.982440    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:32:39.982790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:39.982458    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:32:39.982790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:39.982487    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:33:11.982466505 +0000 UTC m=+98.214072848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:39.982790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:39.982521    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:33:11.982510369 +0000 UTC m=+98.214116715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:32:40.083431 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.083389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:32:40.083431 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.083434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:40.085940 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.085922    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 18:32:40.086058 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.086039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 18:32:40.094058 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:40.094041    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 18:32:40.094121 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:32:40.094111    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:33:44.094094928 +0000 UTC m=+130.325701268 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : secret "metrics-daemon-secret" not found
Sep 09 18:32:40.097037 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.097010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c45724fc-0e25-4804-8f07-891fb9b57ebf-original-pull-secret\") pod \"global-pull-secret-syncer-9hhwv\" (UID: \"c45724fc-0e25-4804-8f07-891fb9b57ebf\") " pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:40.183947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.183906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:40.186308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.186281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hhwv"
Sep 09 18:32:40.186452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.186434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 18:32:40.196246 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.196220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 18:32:40.207727 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.207688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s5c\" (UniqueName: \"kubernetes.io/projected/ead1af0a-36bf-4fa6-9020-a73b9f4a8312-kube-api-access-49s5c\") pod \"network-check-target-tp4fk\" (UID: \"ead1af0a-36bf-4fa6-9020-a73b9f4a8312\") " pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:40.393890 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.393860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdbl9\""
Sep 09 18:32:40.401859 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.401840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:40.455576 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.455540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hhwv"]
Sep 09 18:32:40.462429 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:40.462402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45724fc_0e25_4804_8f07_891fb9b57ebf.slice/crio-b09a00f43a1945abedf16f35d0282ba62e8c6e80ecc0631d86e8bd3e4c60522a WatchSource:0}: Error finding container b09a00f43a1945abedf16f35d0282ba62e8c6e80ecc0631d86e8bd3e4c60522a: Status 404 returned error can't find the container with id b09a00f43a1945abedf16f35d0282ba62e8c6e80ecc0631d86e8bd3e4c60522a
Sep 09 18:32:40.554694 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.554614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hhwv" event={"ID":"c45724fc-0e25-4804-8f07-891fb9b57ebf","Type":"ContainerStarted","Data":"b09a00f43a1945abedf16f35d0282ba62e8c6e80ecc0631d86e8bd3e4c60522a"}
Sep 09 18:32:40.669099 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:40.669066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tp4fk"]
Sep 09 18:32:40.671785 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:32:40.671757    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead1af0a_36bf_4fa6_9020_a73b9f4a8312.slice/crio-3018444aedc6de7cb297b54561d6971f03d81d8e6d5f2b26a7e50e957c0871f6 WatchSource:0}: Error finding container 3018444aedc6de7cb297b54561d6971f03d81d8e6d5f2b26a7e50e957c0871f6: Status 404 returned error can't find the container with id 3018444aedc6de7cb297b54561d6971f03d81d8e6d5f2b26a7e50e957c0871f6
Sep 09 18:32:41.561563 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:41.561490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tp4fk" event={"ID":"ead1af0a-36bf-4fa6-9020-a73b9f4a8312","Type":"ContainerStarted","Data":"3018444aedc6de7cb297b54561d6971f03d81d8e6d5f2b26a7e50e957c0871f6"}
Sep 09 18:32:42.946964 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:42.946926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:42.999058 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:42.999017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:46.578642 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:46.578542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hhwv" event={"ID":"c45724fc-0e25-4804-8f07-891fb9b57ebf","Type":"ContainerStarted","Data":"4134653834fb7919e34d9f84d68669da7ca9c50d9f4fe955ee79d7706c44e313"}
Sep 09 18:32:46.581619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:46.581581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tp4fk" event={"ID":"ead1af0a-36bf-4fa6-9020-a73b9f4a8312","Type":"ContainerStarted","Data":"2bd38f7dbab5cae338fa03b2e3858a496b16a75afa1fa78ff8f33a462dd80cf3"}
Sep 09 18:32:46.581769 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:46.581650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:32:46.594082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:46.594039    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9hhwv" podStartSLOduration=66.732253152 podStartE2EDuration="1m12.594025656s" totalImagesPullingTime="5.861772504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:46.592940496 +0000 UTC m=+72.824546858" watchObservedRunningTime="2026-09-09 18:32:46.594025656 +0000 UTC m=+72.825632017"
Sep 09 18:32:46.606247 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:46.606210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tp4fk" podStartSLOduration=66.944536763 podStartE2EDuration="1m12.60619745s" totalImagesPullingTime="5.661660687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:46.605631967 +0000 UTC m=+72.837238329" watchObservedRunningTime="2026-09-09 18:32:46.60619745 +0000 UTC m=+72.837804021"
Sep 09 18:32:48.277491 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:48.277454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:54.619906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:54.619872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:54.720541 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:54.720507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:32:54.821429 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:32:54.821400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:33:12.013022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:33:12.012926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:33:12.013022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:33:12.012987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:33:12.013479 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:12.013089    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:33:12.013479 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:12.013098    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:33:12.013479 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:12.013114    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:33:12.014167 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:12.014128    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:34:16.013165183 +0000 UTC m=+162.244771526 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:33:12.014278 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:12.014200    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:34:16.014170546 +0000 UTC m=+162.245776886 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:33:14.616015 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:33:14.615981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:33:17.587330 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:33:17.587296    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tp4fk"
Sep 09 18:33:44.144574 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:33:44.144519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:33:44.145223 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:44.144683    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 18:33:44.145223 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:33:44.144801    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs podName:53f382bc-f02f-4c4c-b843-abba4b52c530 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:46.144778026 +0000 UTC m=+252.376384370 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs") pod "network-metrics-daemon-qkdph" (UID: "53f382bc-f02f-4c4c-b843-abba4b52c530") : secret "metrics-daemon-secret" not found
Sep 09 18:34:11.132263 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:11.132217    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" podUID="f2d2846a-4764-40a5-9680-7d8507af6759"
Sep 09 18:34:11.152662 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:11.152623    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vf9rh" podUID="0f3173bc-3ab6-4b3b-889f-8c95808239f7"
Sep 09 18:34:11.287569 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:11.287514    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qkdph" podUID="53f382bc-f02f-4c4c-b843-abba4b52c530"
Sep 09 18:34:11.824780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:11.824754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:34:11.824780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:11.824788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:34:16.068307 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:16.068255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:34:16.068307 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:16.068314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:34:16.068784 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:16.068408    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:34:16.068784 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:16.068419    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 18:34:16.068784 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:16.068435    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5656bf5c48-z49tb: secret "image-registry-tls" not found
Sep 09 18:34:16.068784 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:16.068486    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls podName:0f3173bc-3ab6-4b3b-889f-8c95808239f7 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:18.06847091 +0000 UTC m=+284.300077251 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls") pod "dns-default-vf9rh" (UID: "0f3173bc-3ab6-4b3b-889f-8c95808239f7") : secret "dns-default-metrics-tls" not found
Sep 09 18:34:16.068784 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:34:16.068500    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls podName:f2d2846a-4764-40a5-9680-7d8507af6759 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:18.068494111 +0000 UTC m=+284.300100451 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls") pod "image-registry-5656bf5c48-z49tb" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759") : secret "image-registry-tls" not found
Sep 09 18:34:18.479078 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.479033    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" podUID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 09 18:34:18.847877 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.847794    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f7b4890-b6fb-4fe4-8705-225643b2f36e" containerID="9c82d0d779a696117126264342f47236ffd787ef0588217592403a2bb8ed6a37" exitCode=255
Sep 09 18:34:18.848036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.847874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerDied","Data":"9c82d0d779a696117126264342f47236ffd787ef0588217592403a2bb8ed6a37"}
Sep 09 18:34:18.848206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.848191    2581 scope.go:117] "RemoveContainer" containerID="9c82d0d779a696117126264342f47236ffd787ef0588217592403a2bb8ed6a37"
Sep 09 18:34:18.849463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.849445    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831" containerID="190eb2290b78b24912239d7ebbafcc4e005fb5de4079bd5d3c0fb97ea6275045" exitCode=1
Sep 09 18:34:18.849514 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.849490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerDied","Data":"190eb2290b78b24912239d7ebbafcc4e005fb5de4079bd5d3c0fb97ea6275045"}
Sep 09 18:34:18.849749 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:18.849734    2581 scope.go:117] "RemoveContainer" containerID="190eb2290b78b24912239d7ebbafcc4e005fb5de4079bd5d3c0fb97ea6275045"
Sep 09 18:34:19.855167 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:19.855119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerStarted","Data":"a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132"}
Sep 09 18:34:19.856787 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:19.856762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerStarted","Data":"2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"}
Sep 09 18:34:19.857075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:19.857057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:34:19.857664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:19.857647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:34:23.278277 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:34:23.278237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:35:19.037630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:19.037599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:19.091880 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:19.091856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:22.038043 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.038007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:22.088335 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.088304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:22.850197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.850161    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-czlqj"]
Sep 09 18:35:22.852648 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.852629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.854848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.854825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 18:35:22.855104 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.855085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nn2fg\""
Sep 09 18:35:22.855158 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.855124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 18:35:22.855863 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.855848    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 18:35:22.855961 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.855862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 18:35:22.864747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.864701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czlqj"]
Sep 09 18:35:22.877684 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.877661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:22.931096 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.931065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.931206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.931117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.931206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.931186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gd9\" (UniqueName: \"kubernetes.io/projected/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-api-access-s7gd9\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.931280 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.931215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96041057-1b11-44da-b6ba-21c0425d7b3f-data-volume\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.931280 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.931237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96041057-1b11-44da-b6ba-21c0425d7b3f-crio-socket\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:22.938202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:22.938172    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:23.032552 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.032732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gd9\" (UniqueName: \"kubernetes.io/projected/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-api-access-s7gd9\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.032732 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.032679    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.032732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96041057-1b11-44da-b6ba-21c0425d7b3f-data-volume\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.032913 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96041057-1b11-44da-b6ba-21c0425d7b3f-crio-socket\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.032913 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.032771    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:23.532747172 +0000 UTC m=+229.764353542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.032913 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96041057-1b11-44da-b6ba-21c0425d7b3f-crio-socket\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.032913 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.032847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.033048 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.032975    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.033048 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.033008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96041057-1b11-44da-b6ba-21c0425d7b3f-data-volume\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.033108 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.033100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:23.533084562 +0000 UTC m=+229.764690903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.040307 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.040273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gd9\" (UniqueName: \"kubernetes.io/projected/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-api-access-s7gd9\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.537403 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.537349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.537590 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.537425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:23.537590 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.537472    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.537590 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.537546    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:24.537530961 +0000 UTC m=+230.769137301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.537590 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.537559    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.537799 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:23.537691    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:24.537672879 +0000 UTC m=+230.769279223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.724842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.724805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:23.821033 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.820941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:23.921303 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.921272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:23.986777 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:23.986751    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:24.061490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.061461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:24.453468 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.453429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:24.545686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.545649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:24.545906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.545702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:24.545906 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:24.545798    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:24.545906 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:24.545873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.545856586 +0000 UTC m=+232.777462926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:24.546072 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:24.545919    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:24.546072 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:24.545998    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.545980067 +0000 UTC m=+232.777586424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:24.556012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.555982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:24.620900 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:24.620878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h868k_549f102e-c718-4cdb-8a92-392846408a79/dns-node-resolver/0.log"
Sep 09 18:35:25.228046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:25.228011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:25.277147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:25.277121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:25.621592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:25.621572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2b2wp_9648bc4f-28e2-4501-b9d7-f8d04caa8550/node-ca/0.log"
Sep 09 18:35:26.086882 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.086793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:26.135859 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.135834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:26.381298 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.381272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:26.422204 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.422174    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qm8n4_80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3/serve-healthcheck-canary/0.log"
Sep 09 18:35:26.435641 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.435614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:26.559833 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.559790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:26.560001 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.559848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:26.560001 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:26.559915    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:26.560001 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:26.559963    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:26.560001 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:26.559978    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:30.559962657 +0000 UTC m=+236.791568997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:26.560136 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:26.560009    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:30.559992497 +0000 UTC m=+236.791598858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:26.917329 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:26.917294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:29.686809 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:29.686765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:29.786478 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:29.786444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:29.892340 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:29.892304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:29.988383 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:29.988264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:30.125040 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:30.125003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:30.299332 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:30.299255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:30.591131 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:30.591024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:30.591131 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:30.591096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:30.591348 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:30.591144    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:30.591348 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:30.591220    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:38.591205118 +0000 UTC m=+244.822811458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:30.591348 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:30.591238    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:30.591348 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:30.591300    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:38.591287317 +0000 UTC m=+244.822893657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:30.897766 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:30.897730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:31.397123 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:31.397089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:32.095989 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:32.095959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:32.502952 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:32.502914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:32.730592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:32.730556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:32.906089 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:32.906056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:33.095380 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:33.095339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:35.879442 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:35.879407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:35.981207 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:35.981174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.080448 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.080415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.297028 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.296943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.498864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.498828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.554028 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.553952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.877726 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.877674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:36.934548 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:36.934519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:37.094425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:37.094394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:38.096750 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.096695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:38.498181 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.498149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:38.659516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.659478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:38.659740 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.659584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:38.659740 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:38.659624    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:38.659740 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:35:38.659687    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls podName:96041057-1b11-44da-b6ba-21c0425d7b3f nodeName:}" failed. No retries permitted until 2026-09-09 18:35:54.659671542 +0000 UTC m=+260.891277882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czlqj" (UID: "96041057-1b11-44da-b6ba-21c0425d7b3f") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:38.660133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.660115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96041057-1b11-44da-b6ba-21c0425d7b3f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:38.694073 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.694042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:38.894969 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:38.894905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:39.100644 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:39.100613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:39.901466 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:39.901435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:40.098355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:40.098318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:40.296060 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:40.295979    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:46.220607 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.220568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:35:46.223040 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.223015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f382bc-f02f-4c4c-b843-abba4b52c530-metrics-certs\") pod \"network-metrics-daemon-qkdph\" (UID: \"53f382bc-f02f-4c4c-b843-abba4b52c530\") " pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:35:46.378604 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.378572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:46.380923 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.380904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2784s\""
Sep 09 18:35:46.389319 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.389295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkdph"
Sep 09 18:35:46.427643 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.427588    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:46.655627 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:46.655598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkdph"]
Sep 09 18:35:46.658837 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:35:46.658809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f382bc_f02f_4c4c_b843_abba4b52c530.slice/crio-ea906098e4b2dd29f31b020e84e2a4569c22893e99f870df475d7370c1b92802 WatchSource:0}: Error finding container ea906098e4b2dd29f31b020e84e2a4569c22893e99f870df475d7370c1b92802: Status 404 returned error can't find the container with id ea906098e4b2dd29f31b020e84e2a4569c22893e99f870df475d7370c1b92802
Sep 09 18:35:47.113666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:47.113631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkdph" event={"ID":"53f382bc-f02f-4c4c-b843-abba4b52c530","Type":"ContainerStarted","Data":"ea906098e4b2dd29f31b020e84e2a4569c22893e99f870df475d7370c1b92802"}
Sep 09 18:35:49.121982 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:49.121946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkdph" event={"ID":"53f382bc-f02f-4c4c-b843-abba4b52c530","Type":"ContainerStarted","Data":"fcf80885ddd5841cf9209cb5c9e1720120c7d3df2b23232479c127c9bbb3fde2"}
Sep 09 18:35:49.122379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:49.122027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkdph" event={"ID":"53f382bc-f02f-4c4c-b843-abba4b52c530","Type":"ContainerStarted","Data":"4118364f8af9acec7b208342f4d97bc2d094bdb27806faff0a1db380e0c2e4d4"}
Sep 09 18:35:49.135878 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:49.135804    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qkdph" podStartSLOduration=253.742964424 podStartE2EDuration="4m15.135786208s" totalImagesPullingTime="1.392821784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:49.135233805 +0000 UTC m=+255.366840166" watchObservedRunningTime="2026-09-09 18:35:49.135786208 +0000 UTC m=+255.367392571"
Sep 09 18:35:50.315197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:50.315160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:50.409857 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:50.409828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:53.511214 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:53.511181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:53.607555 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:53.607525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:53.708386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:53.708357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:54.152396 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:54.152366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:54.684331 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:54.684297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:54.686782 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:54.686751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96041057-1b11-44da-b6ba-21c0425d7b3f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czlqj\" (UID: \"96041057-1b11-44da-b6ba-21c0425d7b3f\") " pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:54.965122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:54.965015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nn2fg\""
Sep 09 18:35:54.973111 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:54.973082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czlqj"
Sep 09 18:35:55.245274 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:55.241897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czlqj"]
Sep 09 18:35:55.249052 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:35:55.249022    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96041057_1b11_44da_b6ba_21c0425d7b3f.slice/crio-e8b73bd00c85d6aa73f8f7237252daf820a8de9e6a773f002f98bcb2a76bb74a WatchSource:0}: Error finding container e8b73bd00c85d6aa73f8f7237252daf820a8de9e6a773f002f98bcb2a76bb74a: Status 404 returned error can't find the container with id e8b73bd00c85d6aa73f8f7237252daf820a8de9e6a773f002f98bcb2a76bb74a
Sep 09 18:35:56.144281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:56.144238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czlqj" event={"ID":"96041057-1b11-44da-b6ba-21c0425d7b3f","Type":"ContainerStarted","Data":"6cca58c11b014d705e5f0b41696953d2001a2cacde0c16a5ffb5ed42f9c6e0aa"}
Sep 09 18:35:56.144765 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:56.144302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czlqj" event={"ID":"96041057-1b11-44da-b6ba-21c0425d7b3f","Type":"ContainerStarted","Data":"e8b73bd00c85d6aa73f8f7237252daf820a8de9e6a773f002f98bcb2a76bb74a"}
Sep 09 18:35:57.148967 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:57.148937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czlqj" event={"ID":"96041057-1b11-44da-b6ba-21c0425d7b3f","Type":"ContainerStarted","Data":"9eaa7495e99fd57ce7799e212ce16966ad01a8198fced041ee89192898913d62"}
Sep 09 18:35:57.500036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:57.499962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:57.600387 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:57.600346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:59.157310 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:59.157265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czlqj" event={"ID":"96041057-1b11-44da-b6ba-21c0425d7b3f","Type":"ContainerStarted","Data":"b31d1bcff0200d212b97ec9402a68d21a295309eac08689e85029e734cd9066d"}
Sep 09 18:35:59.173293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:59.173242    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-czlqj" podStartSLOduration=34.1467187 podStartE2EDuration="37.173227804s" totalImagesPullingTime="3.026509104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:35:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:59.172096428 +0000 UTC m=+265.403702792" watchObservedRunningTime="2026-09-09 18:35:59.173227804 +0000 UTC m=+265.404834144"
Sep 09 18:35:59.730823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:59.730778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:59.792097 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:59.792060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:35:59.854254 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:35:59.854227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:01.878452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:01.878412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:03.842658 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:03.842602    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.061450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.061401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.105390 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.105359    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rm4hn"]
Sep 09 18:36:04.108715 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.108684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.111232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.111210    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 18:36:04.111366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.111210    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rcsch\""
Sep 09 18:36:04.112438 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.112418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 18:36:04.112589 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.112416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 18:36:04.112589 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.112421    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 18:36:04.112589 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.112456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 18:36:04.112589 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.112512    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 18:36:04.146232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.146202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.153468 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-sys\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-textfile\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153633 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153633 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-wtmp\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153746 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-metrics-client-ca\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153746 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-root\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgv5k\" (UniqueName: \"kubernetes.io/projected/2683c1ed-ef49-49b5-a980-7c940654fce1-kube-api-access-vgv5k\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.153830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.153789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.252588 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.252556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.254450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgv5k\" (UniqueName: \"kubernetes.io/projected/2683c1ed-ef49-49b5-a980-7c940654fce1-kube-api-access-vgv5k\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-sys\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-textfile\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-wtmp\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-sys\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:04.254581    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-metrics-client-ca\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254646 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:04.254637    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls podName:2683c1ed-ef49-49b5-a980-7c940654fce1 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:04.754619081 +0000 UTC m=+270.986225428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls") pod "node-exporter-rm4hn" (UID: "2683c1ed-ef49-49b5-a980-7c940654fce1") : secret "node-exporter-tls" not found
Sep 09 18:36:04.254951 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-root\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254951 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-wtmp\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254951 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2683c1ed-ef49-49b5-a980-7c940654fce1-root\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.254951 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.254894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-textfile\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.255194 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.255177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-metrics-client-ca\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.255277 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.255256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.257045 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.257025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.262768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.262747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgv5k\" (UniqueName: \"kubernetes.io/projected/2683c1ed-ef49-49b5-a980-7c940654fce1-kube-api-access-vgv5k\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.353478 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.353430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.455419 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.455340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.556183 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.556156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.655290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.655257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:04.759379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.759294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:04.761661 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:04.761639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2683c1ed-ef49-49b5-a980-7c940654fce1-node-exporter-tls\") pod \"node-exporter-rm4hn\" (UID: \"2683c1ed-ef49-49b5-a980-7c940654fce1\") " pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:05.018391 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:05.018308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rm4hn"
Sep 09 18:36:05.028921 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:36:05.028893    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2683c1ed_ef49_49b5_a980_7c940654fce1.slice/crio-a35bf08805503cf376818b8619ad5c1fd34acc628f767f3ac92f6e1df758867b WatchSource:0}: Error finding container a35bf08805503cf376818b8619ad5c1fd34acc628f767f3ac92f6e1df758867b: Status 404 returned error can't find the container with id a35bf08805503cf376818b8619ad5c1fd34acc628f767f3ac92f6e1df758867b
Sep 09 18:36:05.073282 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:05.073219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:05.143530 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:05.143473    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:05.176902 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:05.176870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rm4hn" event={"ID":"2683c1ed-ef49-49b5-a980-7c940654fce1","Type":"ContainerStarted","Data":"a35bf08805503cf376818b8619ad5c1fd34acc628f767f3ac92f6e1df758867b"}
Sep 09 18:36:05.381471 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:05.381439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:06.181683 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:06.181650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rm4hn" event={"ID":"2683c1ed-ef49-49b5-a980-7c940654fce1","Type":"ContainerStarted","Data":"c69a59b80be92f546243e5c1a7b6f1758f3655a77e8bd82a7ed03b788f706dba"}
Sep 09 18:36:07.024387 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.024357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:07.072929 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.072899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:07.189912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.189873    2581 generic.go:356] "Generic (PLEG): container finished" podID="2683c1ed-ef49-49b5-a980-7c940654fce1" containerID="c69a59b80be92f546243e5c1a7b6f1758f3655a77e8bd82a7ed03b788f706dba" exitCode=0
Sep 09 18:36:07.190269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.189937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rm4hn" event={"ID":"2683c1ed-ef49-49b5-a980-7c940654fce1","Type":"ContainerDied","Data":"c69a59b80be92f546243e5c1a7b6f1758f3655a77e8bd82a7ed03b788f706dba"}
Sep 09 18:36:07.271892 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.271862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:07.372699 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.372666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:07.473117 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.473081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:07.574898 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:07.574819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:08.195039 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:08.194999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rm4hn" event={"ID":"2683c1ed-ef49-49b5-a980-7c940654fce1","Type":"ContainerStarted","Data":"b9fc7f4827f9c16ff73b9aa358a56f9f44d25af06716b8df610556eb5155ea5b"}
Sep 09 18:36:08.195431 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:08.195062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rm4hn" event={"ID":"2683c1ed-ef49-49b5-a980-7c940654fce1","Type":"ContainerStarted","Data":"ef070517098b940c74cf77ce2cb64dd3171bee2fa0ef356932fe16ec0e4b3902"}
Sep 09 18:36:08.213595 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:08.213551    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rm4hn" podStartSLOduration=3.131383317 podStartE2EDuration="4.213535019s" totalImagesPullingTime="1.013532699s" totalInitContainerRuntime="68.619003ms" isStatefulPod=true podCreationTimestamp="2026-09-09 18:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:08.212923309 +0000 UTC m=+274.444529673" watchObservedRunningTime="2026-09-09 18:36:08.213535019 +0000 UTC m=+274.445141378"
Sep 09 18:36:08.864495 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:08.864464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:09.069966 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:09.069933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:10.271262 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:10.271214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:11.386105 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:11.386069    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" podUID="1f406003-51a2-4e47-84ae-0debd381f555" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 18:36:14.825442 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:14.825397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vf9rh" podUID="0f3173bc-3ab6-4b3b-889f-8c95808239f7"
Sep 09 18:36:14.825442 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:14.825425    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" podUID="f2d2846a-4764-40a5-9680-7d8507af6759"
Sep 09 18:36:15.217095 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:15.217058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:15.217286 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:15.217067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:16.370507 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:16.370473    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:16.444792 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:16.444748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.023182 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.023152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.069443 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.069409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:18.069613 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.069450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:18.071912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.071891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f3173bc-3ab6-4b3b-889f-8c95808239f7-metrics-tls\") pod \"dns-default-vf9rh\" (UID: \"0f3173bc-3ab6-4b3b-889f-8c95808239f7\") " pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:18.072147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.072126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"image-registry-5656bf5c48-z49tb\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") " pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:18.076047 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.076023    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.220005 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.219970    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-74w6f\""
Sep 09 18:36:18.220864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.220846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgjhb\""
Sep 09 18:36:18.228805 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.228768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:18.228942 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.228847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:18.238186 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.238150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.294146 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.294067    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.342753 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.342703    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.443037 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.443009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:18.508910 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.508881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vf9rh"]
Sep 09 18:36:18.511760 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:18.511656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:36:18.512287 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:36:18.512197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3173bc_3ab6_4b3b_889f_8c95808239f7.slice/crio-0f68d51cce6164450a248e2c2605c8e3e3cfcc55839202abcaa27dbc5ab31d59 WatchSource:0}: Error finding container 0f68d51cce6164450a248e2c2605c8e3e3cfcc55839202abcaa27dbc5ab31d59: Status 404 returned error can't find the container with id 0f68d51cce6164450a248e2c2605c8e3e3cfcc55839202abcaa27dbc5ab31d59
Sep 09 18:36:18.514047 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:36:18.514026    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d2846a_4764_40a5_9680_7d8507af6759.slice/crio-31bce276f73d9e34ca40cfa03fb4257af4cc15b3cb18e997ea309cbe616b3ed2 WatchSource:0}: Error finding container 31bce276f73d9e34ca40cfa03fb4257af4cc15b3cb18e997ea309cbe616b3ed2: Status 404 returned error can't find the container with id 31bce276f73d9e34ca40cfa03fb4257af4cc15b3cb18e997ea309cbe616b3ed2
Sep 09 18:36:19.232780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.232678    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831" containerID="2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c" exitCode=1
Sep 09 18:36:19.232780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.232753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerDied","Data":"2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"}
Sep 09 18:36:19.233248 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.232848    2581 scope.go:117] "RemoveContainer" containerID="190eb2290b78b24912239d7ebbafcc4e005fb5de4079bd5d3c0fb97ea6275045"
Sep 09 18:36:19.233248 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.233150    2581 scope.go:117] "RemoveContainer" containerID="2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"
Sep 09 18:36:19.233385 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:19.233355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-dc5676f5b-xvt94_open-cluster-management-agent-addon(d8da06e8-50eb-41b1-9b5e-c7f722ad7831)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" podUID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831"
Sep 09 18:36:19.234265 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.234010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vf9rh" event={"ID":"0f3173bc-3ab6-4b3b-889f-8c95808239f7","Type":"ContainerStarted","Data":"0f68d51cce6164450a248e2c2605c8e3e3cfcc55839202abcaa27dbc5ab31d59"}
Sep 09 18:36:19.235473 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.235454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" event={"ID":"f2d2846a-4764-40a5-9680-7d8507af6759","Type":"ContainerStarted","Data":"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"}
Sep 09 18:36:19.235592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.235487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" event={"ID":"f2d2846a-4764-40a5-9680-7d8507af6759","Type":"ContainerStarted","Data":"31bce276f73d9e34ca40cfa03fb4257af4cc15b3cb18e997ea309cbe616b3ed2"}
Sep 09 18:36:19.235592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.235532    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:19.237311 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.237290    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f7b4890-b6fb-4fe4-8705-225643b2f36e" containerID="a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132" exitCode=255
Sep 09 18:36:19.237412 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.237333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerDied","Data":"a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132"}
Sep 09 18:36:19.237619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.237605    2581 scope.go:117] "RemoveContainer" containerID="a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132"
Sep 09 18:36:19.237794 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:19.237776    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-658778bc76-7phwf_open-cluster-management-agent-addon(9f7b4890-b6fb-4fe4-8705-225643b2f36e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" podUID="9f7b4890-b6fb-4fe4-8705-225643b2f36e"
Sep 09 18:36:19.243266 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.243250    2581 scope.go:117] "RemoveContainer" containerID="9c82d0d779a696117126264342f47236ffd787ef0588217592403a2bb8ed6a37"
Sep 09 18:36:19.266042 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.266002    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" podStartSLOduration=285.265987922 podStartE2EDuration="4m45.265987922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:19.265188111 +0000 UTC m=+285.496794495" watchObservedRunningTime="2026-09-09 18:36:19.265987922 +0000 UTC m=+285.497594285"
Sep 09 18:36:19.857376 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:19.857338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:36:20.243449 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:20.243370    2581 scope.go:117] "RemoveContainer" containerID="2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"
Sep 09 18:36:20.243886 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:20.243532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-dc5676f5b-xvt94_open-cluster-management-agent-addon(d8da06e8-50eb-41b1-9b5e-c7f722ad7831)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" podUID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831"
Sep 09 18:36:21.248082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.247998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vf9rh" event={"ID":"0f3173bc-3ab6-4b3b-889f-8c95808239f7","Type":"ContainerStarted","Data":"ad10303e74e06eb0530a20da6f75942d82856b6fe1fba703a3b880a5ff8d3c9f"}
Sep 09 18:36:21.248082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.248052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vf9rh" event={"ID":"0f3173bc-3ab6-4b3b-889f-8c95808239f7","Type":"ContainerStarted","Data":"35786049271d1880e662666280ad16e5c457a9b33e2aca96fdf759d1ee03a359"}
Sep 09 18:36:21.248462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.248086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:21.264194 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.264146    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vf9rh" podStartSLOduration=250.820145499 podStartE2EDuration="4m13.264125809s" totalImagesPullingTime="2.44398031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:21.263023647 +0000 UTC m=+287.494630019" watchObservedRunningTime="2026-09-09 18:36:21.264125809 +0000 UTC m=+287.495732170"
Sep 09 18:36:21.368115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.368084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf"
Sep 09 18:36:21.368394 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.368381    2581 scope.go:117] "RemoveContainer" containerID="a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132"
Sep 09 18:36:21.368548 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:21.368530    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-658778bc76-7phwf_open-cluster-management-agent-addon(9f7b4890-b6fb-4fe4-8705-225643b2f36e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" podUID="9f7b4890-b6fb-4fe4-8705-225643b2f36e"
Sep 09 18:36:21.377030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.377011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:36:21.377326 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.377313    2581 scope.go:117] "RemoveContainer" containerID="2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"
Sep 09 18:36:21.377452 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:36:21.377436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-dc5676f5b-xvt94_open-cluster-management-agent-addon(d8da06e8-50eb-41b1-9b5e-c7f722ad7831)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" podUID="d8da06e8-50eb-41b1-9b5e-c7f722ad7831"
Sep 09 18:36:21.386385 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:21.386356    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" podUID="1f406003-51a2-4e47-84ae-0debd381f555" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 18:36:22.101999 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:22.101967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:22.149291 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:22.149264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:29.848964 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:29.848923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:29.899770 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:29.899735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:31.253755 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:31.253698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vf9rh"
Sep 09 18:36:31.385812 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:31.385764    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" podUID="1f406003-51a2-4e47-84ae-0debd381f555" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 18:36:31.385965 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:31.385843    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98"
Sep 09 18:36:31.386282 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:31.386254    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a41a46992159c07449a8e40d71ffe7a4966cdb19a6f697a40c73019ebd488ba7"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 18:36:31.386327 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:31.386301    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" podUID="1f406003-51a2-4e47-84ae-0debd381f555" containerName="service-proxy" containerID="cri-o://a41a46992159c07449a8e40d71ffe7a4966cdb19a6f697a40c73019ebd488ba7" gracePeriod=30
Sep 09 18:36:32.284213 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:32.284174    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f406003-51a2-4e47-84ae-0debd381f555" containerID="a41a46992159c07449a8e40d71ffe7a4966cdb19a6f697a40c73019ebd488ba7" exitCode=2
Sep 09 18:36:32.284611 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:32.284237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerDied","Data":"a41a46992159c07449a8e40d71ffe7a4966cdb19a6f697a40c73019ebd488ba7"}
Sep 09 18:36:32.284611 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:32.284280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-574b4cdf6f-w8l98" event={"ID":"1f406003-51a2-4e47-84ae-0debd381f555","Type":"ContainerStarted","Data":"e5b2127f052709414cfe48fd1638711ec618a15d6ba73adaff2859cde6851df4"}
Sep 09 18:36:33.278778 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:33.278753    2581 scope.go:117] "RemoveContainer" containerID="2e409f39a836c3d2e1baf8a31e0405e02f2cae3bacb8a35dc1e1cfae61152f4c"
Sep 09 18:36:33.646355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:33.646328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:34.189677 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:34.189653    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 18:36:34.280105 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:34.279981    2581 scope.go:117] "RemoveContainer" containerID="a2696f121b842413dcc3c0a99f975937c882abcbfa713f400cf599339e681132"
Sep 09 18:36:34.299008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:34.294331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94" event={"ID":"d8da06e8-50eb-41b1-9b5e-c7f722ad7831","Type":"ContainerStarted","Data":"e14ff88a8b3d28217eb31cde1f7125dbbb2e74105932aea3130f601e8e893358"}
Sep 09 18:36:34.299008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:34.294772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:36:34.299008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:34.296126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc5676f5b-xvt94"
Sep 09 18:36:35.299463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:35.299426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-658778bc76-7phwf" event={"ID":"9f7b4890-b6fb-4fe4-8705-225643b2f36e","Type":"ContainerStarted","Data":"2a8b9b7c42a76d44744da2406b522b70a006ee586de9ba6281e72afed556de18"}
Sep 09 18:36:36.622352 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:36.622319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:36.723298 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:36.723261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:38.233490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:38.233457    2581 patch_prober.go:28] interesting pod/image-registry-5656bf5c48-z49tb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:38.233909 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:38.233506    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:38.645463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:38.645432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:38.693586 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:38.693553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:40.248205 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:40.248171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:36:40.371287 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:40.371189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:40.632942 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:40.632904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:36:42.872262 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:42.872217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:43.856800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:43.856723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:43.956723 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:43.956688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:45.849253 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:45.849209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:45.949559 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:45.949533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:46.050338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:46.050306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:46.189602 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:46.189480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:49.656380 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:49.656342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:49.726859 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:49.726821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:50.099996 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:50.099970    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/init-textfile/0.log"
Sep 09 18:36:50.301351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:50.301323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/node-exporter/0.log"
Sep 09 18:36:50.500664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:50.500593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/kube-rbac-proxy/0.log"
Sep 09 18:36:51.523311 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:51.523276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:51.624470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:51.624435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:56.500434 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:56.500405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vf9rh_0f3173bc-3ab6-4b3b-889f-8c95808239f7/dns/0.log"
Sep 09 18:36:56.700227 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:56.700199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vf9rh_0f3173bc-3ab6-4b3b-889f-8c95808239f7/kube-rbac-proxy/0.log"
Sep 09 18:36:56.902348 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:56.902324    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h868k_549f102e-c718-4cdb-8a92-392846408a79/dns-node-resolver/0.log"
Sep 09 18:36:57.644977 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:57.644941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:57.700083 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:57.700056    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qm8n4_80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3/serve-healthcheck-canary/0.log"
Sep 09 18:36:58.035231 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:58.035158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:36:59.156187 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:36:59.156151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:00.258206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:00.258176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:00.589161 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:00.589077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:00.690171 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:00.690138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:00.791554 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:00.791522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:00.892967 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:00.892937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:04.096093 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:04.096054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:04.208448 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:04.208398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:05.652251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:05.652189    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" containerName="registry" containerID="cri-o://2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e" gracePeriod=30
Sep 09 18:37:06.516921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.516893    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:06.572475 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.572438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:06.890475 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.890453    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:37:06.909636 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.909822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.909822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.909822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-828fz\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.909822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.912696 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.912696 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.912696 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.909895    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration\") pod \"f2d2846a-4764-40a5-9680-7d8507af6759\" (UID: \"f2d2846a-4764-40a5-9680-7d8507af6759\") "
Sep 09 18:37:06.912696 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.912593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:06.912922 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.912858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:06.914248 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.914199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:06.915523 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.914579    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:06.915523 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.915231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:06.915523 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.915395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:06.915969 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.915942    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz" (OuterVolumeSpecName: "kube-api-access-828fz") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "kube-api-access-828fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:06.921910 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:06.921882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f2d2846a-4764-40a5-9680-7d8507af6759" (UID: "f2d2846a-4764-40a5-9680-7d8507af6759"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:37:07.010355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010324    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-registry-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010350    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-trusted-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010361    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-image-registry-private-configuration\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010370    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d2846a-4764-40a5-9680-7d8507af6759-ca-trust-extracted\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010379    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-bound-sa-token\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010387    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d2846a-4764-40a5-9680-7d8507af6759-installation-pull-secrets\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010396    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828fz\" (UniqueName: \"kubernetes.io/projected/f2d2846a-4764-40a5-9680-7d8507af6759-kube-api-access-828fz\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.010575 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.010404    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d2846a-4764-40a5-9680-7d8507af6759-registry-certificates\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:37:07.406908 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.406872    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2d2846a-4764-40a5-9680-7d8507af6759" containerID="2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e" exitCode=0
Sep 09 18:37:07.407088 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.406917    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb"
Sep 09 18:37:07.407088 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.406948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" event={"ID":"f2d2846a-4764-40a5-9680-7d8507af6759","Type":"ContainerDied","Data":"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"}
Sep 09 18:37:07.407088 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.406988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5656bf5c48-z49tb" event={"ID":"f2d2846a-4764-40a5-9680-7d8507af6759","Type":"ContainerDied","Data":"31bce276f73d9e34ca40cfa03fb4257af4cc15b3cb18e997ea309cbe616b3ed2"}
Sep 09 18:37:07.407088 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.407007    2581 scope.go:117] "RemoveContainer" containerID="2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"
Sep 09 18:37:07.415213 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.415191    2581 scope.go:117] "RemoveContainer" containerID="2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"
Sep 09 18:37:07.415480 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:37:07.415457    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e\": container with ID starting with 2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e not found: ID does not exist" containerID="2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"
Sep 09 18:37:07.415546 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.415488    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e"} err="failed to get container status \"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e\": rpc error: code = NotFound desc = could not find container \"2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e\": container with ID starting with 2e0f4ff0899434ea6edadc527c286deadb5e669a16e7e7180435b2dc9cc4a61e not found: ID does not exist"
Sep 09 18:37:07.426168 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.426147    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:37:07.429478 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:07.429458    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5656bf5c48-z49tb"]
Sep 09 18:37:08.282893 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:08.282862    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" path="/var/lib/kubelet/pods/f2d2846a-4764-40a5-9680-7d8507af6759/volumes"
Sep 09 18:37:09.825832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:09.825781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:18.682447 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:18.682377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:18.731160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:18.731132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:21.526051 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:21.526014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:21.629046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:21.629010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:23.813984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:23.813936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:23.891274 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:23.891226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:31.973761 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:31.973692    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:32.021920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:32.021891    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:36.525666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:36.525635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:36.722125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:36.722083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:37:43.059326 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:37:43.059287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:14.270569 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:14.270530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:14.321115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:14.321070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:38.472519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:38.472484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:38.524444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:38.524397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:40.311741 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:40.311693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:40.412905 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:40.412870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:40.512930 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:40.512895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:41.095279 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:41.095239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:50.418635 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:50.418598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:50.471790 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:50.471760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:51.319946 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:51.319911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:51.420916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:51.420883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:38:51.474917 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:38:51.474883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:07.344993 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:07.344886    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:07.396893 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:07.396862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:45.865968 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:45.865917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:45.965949 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:45.965905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:54.043289 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:54.043249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:39:54.132937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:39:54.132896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:26.097954 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:26.097865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:27.466596 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:27.466547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:29.917144 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:29.917095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:30.016595 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:30.016559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:33.910727 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:33.910677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:34.033444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:34.033407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:34.170635 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:34.170540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:35.918147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:35.918114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:36.020888 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:36.020857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:36.121844 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:36.121809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:36.294028 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:36.293953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:44.774000 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:44.773965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:44.833605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:44.833574    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:46.031732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.031687    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"]
Sep 09 18:40:46.032198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.032031    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" containerName="registry"
Sep 09 18:40:46.032198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.032046    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" containerName="registry"
Sep 09 18:40:46.032198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.032113    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d2846a-4764-40a5-9680-7d8507af6759" containerName="registry"
Sep 09 18:40:46.034902 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.034880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.037420 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.037396    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:40:46.037517 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.037400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 18:40:46.038321 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.038306    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zcpkj\""
Sep 09 18:40:46.044546 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.044527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"]
Sep 09 18:40:46.176439 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.176408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1ec764-8cfc-4b78-88dc-979542a5baa8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.176616 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.176452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/2a1ec764-8cfc-4b78-88dc-979542a5baa8-kube-api-access-9c7x7\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.277675 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.277647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1ec764-8cfc-4b78-88dc-979542a5baa8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.277837 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.277690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/2a1ec764-8cfc-4b78-88dc-979542a5baa8-kube-api-access-9c7x7\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.278014 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.277995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1ec764-8cfc-4b78-88dc-979542a5baa8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.286249 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.286182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7x7\" (UniqueName: \"kubernetes.io/projected/2a1ec764-8cfc-4b78-88dc-979542a5baa8-kube-api-access-9c7x7\") pod \"openshift-lws-operator-fd8ccff4c-5m9kb\" (UID: \"2a1ec764-8cfc-4b78-88dc-979542a5baa8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.344027 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.343984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"
Sep 09 18:40:46.613572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.613540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb"]
Sep 09 18:40:46.616514 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:40:46.616489    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a1ec764_8cfc_4b78_88dc_979542a5baa8.slice/crio-1fdda23ec26002936032a83aa53542bec35bf3bf869ad0cc3b8d7762c0436937 WatchSource:0}: Error finding container 1fdda23ec26002936032a83aa53542bec35bf3bf869ad0cc3b8d7762c0436937: Status 404 returned error can't find the container with id 1fdda23ec26002936032a83aa53542bec35bf3bf869ad0cc3b8d7762c0436937
Sep 09 18:40:46.618583 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:46.618563    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:40:47.114138 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:47.114101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb" event={"ID":"2a1ec764-8cfc-4b78-88dc-979542a5baa8","Type":"ContainerStarted","Data":"1fdda23ec26002936032a83aa53542bec35bf3bf869ad0cc3b8d7762c0436937"}
Sep 09 18:40:50.126360 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:50.126321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb" event={"ID":"2a1ec764-8cfc-4b78-88dc-979542a5baa8","Type":"ContainerStarted","Data":"0839aacb90d7fa92446d72c7841665ced2f7c16b0865bb87bb106e0f9be0eb47"}
Sep 09 18:40:50.142361 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:50.142308    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5m9kb" podStartSLOduration=1.61072489 podStartE2EDuration="4.142291138s" totalImagesPullingTime="2.531566248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:40:50.14148718 +0000 UTC m=+556.373093541" watchObservedRunningTime="2026-09-09 18:40:50.142291138 +0000 UTC m=+556.373897514"
Sep 09 18:40:50.870608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:50.870579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:50.971747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:50.971703    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:40:52.615528 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:40:52.615494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:08.979915 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:08.979832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:09.162084 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:09.162053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:09.586684 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:09.586610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.028572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.028542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.134683 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.134656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.228656 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.228624    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"]
Sep 09 18:41:10.231772 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.231755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.234496 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.234471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 18:41:10.234614 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.234581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jqmk5\""
Sep 09 18:41:10.234761 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.234745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 18:41:10.234820 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.234810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 18:41:10.234947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.234930    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 18:41:10.246870 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.246846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.246984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.246878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.246984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.246900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.246984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.246952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.246984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.246973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcjr\" (UniqueName: \"kubernetes.io/projected/736fa91d-a021-450a-8b82-52594a11ce3b-kube-api-access-qmcjr\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.263057 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.263027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"]
Sep 09 18:41:10.348007 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.347923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.348007 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.347964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.348007 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.347998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.348283 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.348034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.348283 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.348068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcjr\" (UniqueName: \"kubernetes.io/projected/736fa91d-a021-450a-8b82-52594a11ce3b-kube-api-access-qmcjr\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.348445 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.348411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.350447 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.350419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736fa91d-a021-450a-8b82-52594a11ce3b-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.350569 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.350495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.350631 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.350604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/736fa91d-a021-450a-8b82-52594a11ce3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.361768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.361746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcjr\" (UniqueName: \"kubernetes.io/projected/736fa91d-a021-450a-8b82-52594a11ce3b-kube-api-access-qmcjr\") pod \"opendatahub-operator-controller-manager-5f66f47469-mmsvq\" (UID: \"736fa91d-a021-450a-8b82-52594a11ce3b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.449886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.449844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.541686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.541654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:10.582495 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.582465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.708148 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.708115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:10.839619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.839584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"]
Sep 09 18:41:10.842733 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:10.842692    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736fa91d_a021_450a_8b82_52594a11ce3b.slice/crio-dfebb90f45d4b2ec40a5e6a33b20c0479e5b81ae9053311d44dbfcd27d881485 WatchSource:0}: Error finding container dfebb90f45d4b2ec40a5e6a33b20c0479e5b81ae9053311d44dbfcd27d881485: Status 404 returned error can't find the container with id dfebb90f45d4b2ec40a5e6a33b20c0479e5b81ae9053311d44dbfcd27d881485
Sep 09 18:41:10.911975 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:10.911939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:11.129849 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:11.129816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:11.200078 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:11.200044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq" event={"ID":"736fa91d-a021-450a-8b82-52594a11ce3b","Type":"ContainerStarted","Data":"dfebb90f45d4b2ec40a5e6a33b20c0479e5b81ae9053311d44dbfcd27d881485"}
Sep 09 18:41:11.291584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:11.291549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:16.222293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:16.222204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq" event={"ID":"736fa91d-a021-450a-8b82-52594a11ce3b","Type":"ContainerStarted","Data":"c4d45ee1392302e6fc583ea1a18b5094176e0ab059b189a35888f0fda8adeac7"}
Sep 09 18:41:17.227021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:17.226988    2581 generic.go:356] "Generic (PLEG): container finished" podID="736fa91d-a021-450a-8b82-52594a11ce3b" containerID="c4d45ee1392302e6fc583ea1a18b5094176e0ab059b189a35888f0fda8adeac7" exitCode=0
Sep 09 18:41:17.227430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:17.227089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq" event={"ID":"736fa91d-a021-450a-8b82-52594a11ce3b","Type":"ContainerDied","Data":"c4d45ee1392302e6fc583ea1a18b5094176e0ab059b189a35888f0fda8adeac7"}
Sep 09 18:41:18.232389 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:18.232357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq" event={"ID":"736fa91d-a021-450a-8b82-52594a11ce3b","Type":"ContainerStarted","Data":"06876e1314e6b1ccc409c3e28a4f47495bd0cc43de0ab36fa51eea4ed79466c1"}
Sep 09 18:41:18.232871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:18.232475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:18.261034 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:18.260986    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq" podStartSLOduration=1.727025755 podStartE2EDuration="8.260970728s" totalImagesPullingTime="5.443412821s" totalInitContainerRuntime="1.090532152s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:18.258775728 +0000 UTC m=+584.490382082" watchObservedRunningTime="2026-09-09 18:41:18.260970728 +0000 UTC m=+584.492577089"
Sep 09 18:41:21.365975 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:21.365944    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:21.484029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:21.483990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:29.238848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:29.238816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-mmsvq"
Sep 09 18:41:31.552525 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:31.552493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:31.838599 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:31.838517    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:32.372408 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.372344    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:32.507955 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.507920    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"]
Sep 09 18:41:32.510929 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.510915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.519372 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.519351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 18:41:32.519484 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.519362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 18:41:32.519484 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.519433    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 18:41:32.519677 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.519659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 18:41:32.519771 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.519694    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rlbzg\""
Sep 09 18:41:32.527918 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.527899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"]
Sep 09 18:41:32.602317 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.602284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.602734 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.602324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tmp\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.602734 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.602347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6lm\" (UniqueName: \"kubernetes.io/projected/246d7540-c8fb-4f28-b6cd-b42be3c2539e-kube-api-access-pn6lm\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.703037 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.702943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tmp\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.703037 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.702987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6lm\" (UniqueName: \"kubernetes.io/projected/246d7540-c8fb-4f28-b6cd-b42be3c2539e-kube-api-access-pn6lm\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.703257 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.703045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.705383 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.705351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tmp\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.705544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.705526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246d7540-c8fb-4f28-b6cd-b42be3c2539e-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.721053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.721028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6lm\" (UniqueName: \"kubernetes.io/projected/246d7540-c8fb-4f28-b6cd-b42be3c2539e-kube-api-access-pn6lm\") pod \"kube-auth-proxy-799dfc8bc6-2phn9\" (UID: \"246d7540-c8fb-4f28-b6cd-b42be3c2539e\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:32.725082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.725054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:32.819340 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:32.819311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"
Sep 09 18:41:33.146214 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:33.146189    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d7540_c8fb_4f28_b6cd_b42be3c2539e.slice/crio-330f7142763233e6847b8d604f36ef2e00a480300d3f2986a616947ea66b37e9 WatchSource:0}: Error finding container 330f7142763233e6847b8d604f36ef2e00a480300d3f2986a616947ea66b37e9: Status 404 returned error can't find the container with id 330f7142763233e6847b8d604f36ef2e00a480300d3f2986a616947ea66b37e9
Sep 09 18:41:33.177925 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:33.177895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9"]
Sep 09 18:41:33.282305 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:33.282274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9" event={"ID":"246d7540-c8fb-4f28-b6cd-b42be3c2539e","Type":"ContainerStarted","Data":"330f7142763233e6847b8d604f36ef2e00a480300d3f2986a616947ea66b37e9"}
Sep 09 18:41:34.384034 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:34.383986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:35.605036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.605000    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:35.609584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.609563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.612767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.612736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 18:41:35.612872 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.612736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 18:41:35.613123 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.613066    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 18:41:35.613239 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.613203    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 18:41:35.613239 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.613220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j6l6g\""
Sep 09 18:41:35.620174 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.620149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:35.650065 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.650030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:35.725388 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725577 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725577 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725577 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzjk\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725577 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725798 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.725798 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.725682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826354 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzjk\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.826550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.826498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.827253 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.827126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.827934 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.827908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:35.829043 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.829022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.829141 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.829080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.829204 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.829189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.829410 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.829391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.849857 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.849829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.863423 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.863343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzjk\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk\") pod \"istiod-openshift-gateway-7b7995d5cd-jjhf5\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:35.919342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:35.919307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:36.566726 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:36.566678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 18:41:36.792155 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:36.792078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:36.795157 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:36.795135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed390b1_0188_4255_a28e_da4eb3559df6.slice/crio-e341538490a03a1e3bebe340e46548e5a69b75150ad9e5e240d34938128e3d08 WatchSource:0}: Error finding container e341538490a03a1e3bebe340e46548e5a69b75150ad9e5e240d34938128e3d08: Status 404 returned error can't find the container with id e341538490a03a1e3bebe340e46548e5a69b75150ad9e5e240d34938128e3d08
Sep 09 18:41:37.302533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:37.302493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9" event={"ID":"246d7540-c8fb-4f28-b6cd-b42be3c2539e","Type":"ContainerStarted","Data":"e43245db55abc81eb7a92fb7a759a1f0bbfb51a0d4bc40e21e6f4e3669e78140"}
Sep 09 18:41:37.303742 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:37.303702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" event={"ID":"fed390b1-0188-4255-a28e-da4eb3559df6","Type":"ContainerStarted","Data":"e341538490a03a1e3bebe340e46548e5a69b75150ad9e5e240d34938128e3d08"}
Sep 09 18:41:37.342644 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:37.342539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-2phn9" podStartSLOduration=1.929200866 podStartE2EDuration="5.342521052s" totalImagesPullingTime="3.413320186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:37.342178284 +0000 UTC m=+603.573784645" watchObservedRunningTime="2026-09-09 18:41:37.342521052 +0000 UTC m=+603.574127414"
Sep 09 18:41:38.052102 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:38.052066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:39.872001 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:39.871955    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 18:41:39.872298 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:39.872046    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 18:41:40.318920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.318825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" event={"ID":"fed390b1-0188-4255-a28e-da4eb3559df6","Type":"ContainerStarted","Data":"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"}
Sep 09 18:41:40.319077 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.318977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:40.320581 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.320551    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-jjhf5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 18:41:40.320735 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.320618    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:41:40.366670 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.366616    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" podStartSLOduration=2.292165786 podStartE2EDuration="5.36660004s" totalImagesPullingTime="3.074434254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:40.360641325 +0000 UTC m=+606.592247690" watchObservedRunningTime="2026-09-09 18:41:40.36660004 +0000 UTC m=+606.598206403"
Sep 09 18:41:40.971286 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:40.971245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:41.323126 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:41.323047    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-jjhf5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 18:41:41.323126 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:41.323098    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:41:41.731217 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:41.731175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:42.422643 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:42.422606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:42.892199 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:42.892173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:43.205851 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.205771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94m4h"]
Sep 09 18:41:43.208844 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.208829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.215060 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.215035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 18:41:43.218053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.218035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2g46n\""
Sep 09 18:41:43.261766 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.261721    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94m4h"]
Sep 09 18:41:43.284777 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.284748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvkz\" (UniqueName: \"kubernetes.io/projected/8cb9ee65-0c59-4934-9bff-98575f73271d-kube-api-access-ztvkz\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.284945 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.284789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.385387 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.385345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvkz\" (UniqueName: \"kubernetes.io/projected/8cb9ee65-0c59-4934-9bff-98575f73271d-kube-api-access-ztvkz\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.385387 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.385387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.385630 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:41:43.385518    2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 18:41:43.385630 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:41:43.385587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert podName:8cb9ee65-0c59-4934-9bff-98575f73271d nodeName:}" failed. No retries permitted until 2026-09-09 18:41:43.885565786 +0000 UTC m=+610.117172139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert") pod "odh-model-controller-7998c67dc4-94m4h" (UID: "8cb9ee65-0c59-4934-9bff-98575f73271d") : secret "odh-model-controller-webhook-cert" not found
Sep 09 18:41:43.406105 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.406082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvkz\" (UniqueName: \"kubernetes.io/projected/8cb9ee65-0c59-4934-9bff-98575f73271d-kube-api-access-ztvkz\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.889056 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.889016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:43.891630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:43.891608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9ee65-0c59-4934-9bff-98575f73271d-cert\") pod \"odh-model-controller-7998c67dc4-94m4h\" (UID: \"8cb9ee65-0c59-4934-9bff-98575f73271d\") " pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:44.121098 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:44.121064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:44.324217 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:44.324189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:44.410339 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:44.410312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94m4h"]
Sep 09 18:41:44.414243 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:44.414216    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb9ee65_0c59_4934_9bff_98575f73271d.slice/crio-555039611e80617f9b887eaf409a924bf2497b259a6379dabe32c0e66706c627 WatchSource:0}: Error finding container 555039611e80617f9b887eaf409a924bf2497b259a6379dabe32c0e66706c627: Status 404 returned error can't find the container with id 555039611e80617f9b887eaf409a924bf2497b259a6379dabe32c0e66706c627
Sep 09 18:41:45.339179 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:45.339106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h" event={"ID":"8cb9ee65-0c59-4934-9bff-98575f73271d","Type":"ContainerStarted","Data":"555039611e80617f9b887eaf409a924bf2497b259a6379dabe32c0e66706c627"}
Sep 09 18:41:48.352285 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:48.352247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h" event={"ID":"8cb9ee65-0c59-4934-9bff-98575f73271d","Type":"ContainerStarted","Data":"97a8393e7c736d2bae51e430f8fdfb2d3f1df47960d5a4a72597a48520ab9246"}
Sep 09 18:41:48.352675 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:48.352322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:41:48.386214 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:48.386162    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h" podStartSLOduration=2.470260747 podStartE2EDuration="5.386146957s" totalImagesPullingTime="2.91588621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:48.38474938 +0000 UTC m=+614.616355735" watchObservedRunningTime="2026-09-09 18:41:48.386146957 +0000 UTC m=+614.617753339"
Sep 09 18:41:48.550518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:48.550488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:50.917585 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:50.917548    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"]
Sep 09 18:41:50.924630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:50.924612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.029464 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.029433    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"]
Sep 09 18:41:51.050390 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7x6\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-kube-api-access-9v7x6\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050547 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050547 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050547 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050547 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050547 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.050745 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.050601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1a572-3c6a-4ecc-9dc8-e920942892ce-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151596 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151596 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.151842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1a572-3c6a-4ecc-9dc8-e920942892ce-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.152013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.151872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7x6\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-kube-api-access-9v7x6\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.152374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.152346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.154265 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.154237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1a572-3c6a-4ecc-9dc8-e920942892ce-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.154450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.154426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.154520 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.154428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.154520 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.154436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.180780 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.180700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.199604 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.199569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7x6\" (UniqueName: \"kubernetes.io/projected/00c1a572-3c6a-4ecc-9dc8-e920942892ce-kube-api-access-9v7x6\") pod \"istiod-openshift-gateway-5677ccb49b-r68j7\" (UID: \"00c1a572-3c6a-4ecc-9dc8-e920942892ce\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.233318 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.233291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:51.565309 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.565288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"]
Sep 09 18:41:51.567376 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:51.567350    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c1a572_3c6a_4ecc_9dc8_e920942892ce.slice/crio-105865eafbf795136555008bcb9b60d458c495f856a1d174b10c7313052dac8b WatchSource:0}: Error finding container 105865eafbf795136555008bcb9b60d458c495f856a1d174b10c7313052dac8b: Status 404 returned error can't find the container with id 105865eafbf795136555008bcb9b60d458c495f856a1d174b10c7313052dac8b
Sep 09 18:41:51.570170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.570139    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 18:41:51.570229 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:51.570198    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 18:41:52.369817 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.369778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7" event={"ID":"00c1a572-3c6a-4ecc-9dc8-e920942892ce","Type":"ContainerStarted","Data":"30cbd25c82125235da1e7df7e1763a2800546f855e35acf71fa28804b570c0e1"}
Sep 09 18:41:52.370216 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.369840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:52.370216 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.369859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7" event={"ID":"00c1a572-3c6a-4ecc-9dc8-e920942892ce","Type":"ContainerStarted","Data":"105865eafbf795136555008bcb9b60d458c495f856a1d174b10c7313052dac8b"}
Sep 09 18:41:52.371559 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.371537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7"
Sep 09 18:41:52.476328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.476277    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r68j7" podStartSLOduration=2.476260411 podStartE2EDuration="2.476260411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:52.41149228 +0000 UTC m=+618.643098654" watchObservedRunningTime="2026-09-09 18:41:52.476260411 +0000 UTC m=+618.707866773"
Sep 09 18:41:52.527030 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.527000    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:52.527197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.527178    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery" containerID="cri-o://a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf" gracePeriod=30
Sep 09 18:41:52.762141 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.762119    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:52.867154 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vzjk\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867517 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.867517 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.867355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts\") pod \"fed390b1-0188-4255-a28e-da4eb3559df6\" (UID: \"fed390b1-0188-4255-a28e-da4eb3559df6\") "
Sep 09 18:41:52.868098 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.868045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:41:52.870017 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.869983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk" (OuterVolumeSpecName: "kube-api-access-6vzjk") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "kube-api-access-6vzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:41:52.870471 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.870445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts" (OuterVolumeSpecName: "cacerts") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:41:52.870471 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.870454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:41:52.870686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.870661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token" (OuterVolumeSpecName: "istio-token") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:41:52.870797 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.870776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:41:52.870856 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.870834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs" (OuterVolumeSpecName: "local-certs") pod "fed390b1-0188-4255-a28e-da4eb3559df6" (UID: "fed390b1-0188-4255-a28e-da4eb3559df6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968047    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-kubeconfig\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968073    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vzjk\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-kube-api-access-6vzjk\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968082    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed390b1-0188-4255-a28e-da4eb3559df6-istio-token\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968090    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-dns-cert\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968098    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed390b1-0188-4255-a28e-da4eb3559df6-local-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968107    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fed390b1-0188-4255-a28e-da4eb3559df6-istio-csr-ca-configmap\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:52.968122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:52.968114    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fed390b1-0188-4255-a28e-da4eb3559df6-cacerts\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:41:53.374775 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.374747    2581 generic.go:356] "Generic (PLEG): container finished" podID="fed390b1-0188-4255-a28e-da4eb3559df6" containerID="a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf" exitCode=0
Sep 09 18:41:53.375169 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.374795    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"
Sep 09 18:41:53.375169 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.374832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" event={"ID":"fed390b1-0188-4255-a28e-da4eb3559df6","Type":"ContainerDied","Data":"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"}
Sep 09 18:41:53.375169 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.374889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5" event={"ID":"fed390b1-0188-4255-a28e-da4eb3559df6","Type":"ContainerDied","Data":"e341538490a03a1e3bebe340e46548e5a69b75150ad9e5e240d34938128e3d08"}
Sep 09 18:41:53.375169 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.374914    2581 scope.go:117] "RemoveContainer" containerID="a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"
Sep 09 18:41:53.383150 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.383132    2581 scope.go:117] "RemoveContainer" containerID="a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"
Sep 09 18:41:53.383400 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:41:53.383384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf\": container with ID starting with a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf not found: ID does not exist" containerID="a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"
Sep 09 18:41:53.383451 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.383408    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf"} err="failed to get container status \"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf\": rpc error: code = NotFound desc = could not find container \"a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf\": container with ID starting with a58445b44adc64b2287cb11129bdf270b659683073ec422b9fed2755874c1bbf not found: ID does not exist"
Sep 09 18:41:53.408578 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.408555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:53.414338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:53.414316    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jjhf5"]
Sep 09 18:41:54.286329 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:54.286281    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" path="/var/lib/kubelet/pods/fed390b1-0188-4255-a28e-da4eb3559df6/volumes"
Sep 09 18:41:55.325628 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.325548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:55.676903 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.676873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:55.838629 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.838585    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"]
Sep 09 18:41:55.839039 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.839017    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery"
Sep 09 18:41:55.839148 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.839041    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery"
Sep 09 18:41:55.839148 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.839113    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed390b1-0188-4255-a28e-da4eb3559df6" containerName="discovery"
Sep 09 18:41:55.841895 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.841875    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"]
Sep 09 18:41:55.842005 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.841991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:55.844053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.844035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:55.845648 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.845632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 18:41:55.847787 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.847767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 18:41:55.853848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.853830    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jtxfn\""
Sep 09 18:41:55.855809 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.855575    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-whngp\""
Sep 09 18:41:55.885437 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.885403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"]
Sep 09 18:41:55.886804 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.886773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"]
Sep 09 18:41:55.993950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.993849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:55.993950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.993889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r277f\" (UniqueName: \"kubernetes.io/projected/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-kube-api-access-r277f\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:55.993950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.993912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:55.993950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:55.993936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2ds\" (UniqueName: \"kubernetes.io/projected/a09abe68-df91-42b1-9c06-eafb7fac39b9-kube-api-access-hs2ds\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.094425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.094389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.094425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.094430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r277f\" (UniqueName: \"kubernetes.io/projected/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-kube-api-access-r277f\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:56.094681 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.094450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:56.094681 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.094474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2ds\" (UniqueName: \"kubernetes.io/projected/a09abe68-df91-42b1-9c06-eafb7fac39b9-kube-api-access-hs2ds\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.094681 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:41:56.094537    2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 18:41:56.094681 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:41:56.094605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert podName:a09abe68-df91-42b1-9c06-eafb7fac39b9 nodeName:}" failed. No retries permitted until 2026-09-09 18:41:56.594588504 +0000 UTC m=+622.826194844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert") pod "llmisvc-controller-manager-777c6bd7d5-fxnx2" (UID: "a09abe68-df91-42b1-9c06-eafb7fac39b9") : secret "llmisvc-webhook-server-cert" not found
Sep 09 18:41:56.096991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.096971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-cert\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:56.136824 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.136782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r277f\" (UniqueName: \"kubernetes.io/projected/2686874d-43c6-4140-9f7d-fbe9dbd5ddb9-kube-api-access-r277f\") pod \"kserve-controller-manager-d6f4b65b8-t9lkz\" (UID: \"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:56.137544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.137525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2ds\" (UniqueName: \"kubernetes.io/projected/a09abe68-df91-42b1-9c06-eafb7fac39b9-kube-api-access-hs2ds\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.153544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.153509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:41:56.229636 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.229603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:56.468944 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:56.468916    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2686874d_43c6_4140_9f7d_fbe9dbd5ddb9.slice/crio-b0f3b63b668004f6743986c203f2dd06a3bfe9e2d3bd4035068a8a4229154c0c WatchSource:0}: Error finding container b0f3b63b668004f6743986c203f2dd06a3bfe9e2d3bd4035068a8a4229154c0c: Status 404 returned error can't find the container with id b0f3b63b668004f6743986c203f2dd06a3bfe9e2d3bd4035068a8a4229154c0c
Sep 09 18:41:56.471840 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.471821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"]
Sep 09 18:41:56.598819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.598788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.601362 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.601339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09abe68-df91-42b1-9c06-eafb7fac39b9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fxnx2\" (UID: \"a09abe68-df91-42b1-9c06-eafb7fac39b9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:56.682441 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.682409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:41:56.759375 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:56.759283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:41:57.049012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:57.048877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"]
Sep 09 18:41:57.051313 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:41:57.051289    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda09abe68_df91_42b1_9c06_eafb7fac39b9.slice/crio-9b8f5dfa5f823e7ec1cf777c8544fae55461666e634566fa5610e0902ce3701d WatchSource:0}: Error finding container 9b8f5dfa5f823e7ec1cf777c8544fae55461666e634566fa5610e0902ce3701d: Status 404 returned error can't find the container with id 9b8f5dfa5f823e7ec1cf777c8544fae55461666e634566fa5610e0902ce3701d
Sep 09 18:41:57.391821 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:57.391780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2" event={"ID":"a09abe68-df91-42b1-9c06-eafb7fac39b9","Type":"ContainerStarted","Data":"9b8f5dfa5f823e7ec1cf777c8544fae55461666e634566fa5610e0902ce3701d"}
Sep 09 18:41:57.392980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:57.392955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz" event={"ID":"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9","Type":"ContainerStarted","Data":"b0f3b63b668004f6743986c203f2dd06a3bfe9e2d3bd4035068a8a4229154c0c"}
Sep 09 18:41:59.359627 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:41:59.359587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-94m4h"
Sep 09 18:42:00.408299 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:00.408265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2" event={"ID":"a09abe68-df91-42b1-9c06-eafb7fac39b9","Type":"ContainerStarted","Data":"7015b9f9d8d052396eaabbd7b7687fbc5f3523c77ecb6e349dc64f81e83e7c41"}
Sep 09 18:42:00.408759 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:00.408392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:42:00.441718 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:00.441666    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2" podStartSLOduration=2.552363399 podStartE2EDuration="5.441649408s" totalImagesPullingTime="2.889286009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:42:00.438745529 +0000 UTC m=+626.670351891" watchObservedRunningTime="2026-09-09 18:42:00.441649408 +0000 UTC m=+626.673255769"
Sep 09 18:42:03.423518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:03.423478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz" event={"ID":"2686874d-43c6-4140-9f7d-fbe9dbd5ddb9","Type":"ContainerStarted","Data":"98e22126ef40940ec6dfe3663f1c385dbdb70ea03ca6ae5e390e6dff0f100233"}
Sep 09 18:42:03.423925 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:03.423541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:42:03.487217 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:03.487160    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz" podStartSLOduration=2.282760189 podStartE2EDuration="8.487142335s" totalImagesPullingTime="6.204382146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:42:03.486623097 +0000 UTC m=+629.718229458" watchObservedRunningTime="2026-09-09 18:42:03.487142335 +0000 UTC m=+629.718748697"
Sep 09 18:42:08.853773 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:08.853731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:42:31.416212 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:31.416184    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fxnx2"
Sep 09 18:42:34.432233 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:34.432201    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t9lkz"
Sep 09 18:42:43.770885 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:43.770843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:42:43.976761 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:43.976726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:42:59.780013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:59.779979    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:42:59.934662 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:42:59.934631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:01.899301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:01.899269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:02.217341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:02.217259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:02.711451 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:02.711421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:03.495222 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:03.495190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:03.909921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:03.909887    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:05.658386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:05.658356    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:05.821886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:05.821853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:07.609796 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:07.609754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:08.971552 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:08.971521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:09.903879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:09.903844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:10.123297 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.123266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:10.150930 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.150890    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z7nss"]
Sep 09 18:43:10.153223 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.153203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:10.158927 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.158866    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 18:43:10.161011 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.160992    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 18:43:10.161118 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.161068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wvbcz\""
Sep 09 18:43:10.167497 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.167475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z7nss"]
Sep 09 18:43:10.260985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.260947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pksg\" (UniqueName: \"kubernetes.io/projected/760ec79e-6761-49ba-a4ed-71032df78b0b-kube-api-access-6pksg\") pod \"authorino-operator-76b478bd78-z7nss\" (UID: \"760ec79e-6761-49ba-a4ed-71032df78b0b\") " pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:10.361561 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.361521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pksg\" (UniqueName: \"kubernetes.io/projected/760ec79e-6761-49ba-a4ed-71032df78b0b-kube-api-access-6pksg\") pod \"authorino-operator-76b478bd78-z7nss\" (UID: \"760ec79e-6761-49ba-a4ed-71032df78b0b\") " pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:10.398365 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.398330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pksg\" (UniqueName: \"kubernetes.io/projected/760ec79e-6761-49ba-a4ed-71032df78b0b-kube-api-access-6pksg\") pod \"authorino-operator-76b478bd78-z7nss\" (UID: \"760ec79e-6761-49ba-a4ed-71032df78b0b\") " pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:10.462754 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.462662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:10.597607 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.597576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:10.799610 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:10.799500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-z7nss"]
Sep 09 18:43:10.803450 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:43:10.803423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760ec79e_6761_49ba_a4ed_71032df78b0b.slice/crio-595b6dc7a5c605b10d8353381ee345c132458e9b178b00d9b53dfc8a53b3854d WatchSource:0}: Error finding container 595b6dc7a5c605b10d8353381ee345c132458e9b178b00d9b53dfc8a53b3854d: Status 404 returned error can't find the container with id 595b6dc7a5c605b10d8353381ee345c132458e9b178b00d9b53dfc8a53b3854d
Sep 09 18:43:11.094952 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:11.094867    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:11.692666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:11.692630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss" event={"ID":"760ec79e-6761-49ba-a4ed-71032df78b0b","Type":"ContainerStarted","Data":"595b6dc7a5c605b10d8353381ee345c132458e9b178b00d9b53dfc8a53b3854d"}
Sep 09 18:43:13.146251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:13.146218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:14.625933 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:14.625901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:14.706181 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:14.706150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss" event={"ID":"760ec79e-6761-49ba-a4ed-71032df78b0b","Type":"ContainerStarted","Data":"c90c9e34e606f6c18ad84be4d211d70d3b94d398e4ada6842d6ac9b38543afbf"}
Sep 09 18:43:14.706338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:14.706283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:14.760520 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:14.760464    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss" podStartSLOduration=1.915093292 podStartE2EDuration="4.760447863s" totalImagesPullingTime="2.845354571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:43:14.75839691 +0000 UTC m=+700.990003271" watchObservedRunningTime="2026-09-09 18:43:14.760447863 +0000 UTC m=+700.992054224"
Sep 09 18:43:16.275234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:16.275203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:16.447731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:16.447680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:16.614921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:16.614893    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:18.423366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:18.423332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:18.581378 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:18.581351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:18.819546 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:18.819458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:18.957728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:18.957681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:19.172013 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:19.171982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:19.342826 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:19.342792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:20.798598 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:20.798566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:25.713304 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:25.713231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-z7nss"
Sep 09 18:43:27.301821 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:27.301781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:27.602455 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:27.602424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:27.820179 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:27.820142    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:30.189532 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:30.189501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:30.388418 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:30.388390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:34.411318 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:34.411287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:34.574374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:34.574336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:39.839200 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:39.839165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:41.591912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:41.591852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:41.777377 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:41.777349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:49.010476 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:49.010443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:49.241004 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:49.240969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:49.503232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:49.503198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:51.415093 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:51.415041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:51.949329 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:51.949299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:56.027993 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:56.027960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:43:56.201245 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:43:56.201207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:00.525064 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:00.525033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:01.226425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:01.226391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:02.660877 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:02.660843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:02.780541 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:02.780509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:03.063444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.063343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:03.087285 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.087249    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:03.089982 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.089958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.092998 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.092975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j45lb\""
Sep 09 18:44:03.093310 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.093294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 18:44:03.104914 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.104860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:03.148789 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.148757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:03.162921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.162884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:03.175064 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.175037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48kz\" (UniqueName: \"kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.175156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.175113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.259348 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.259314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:03.276504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.276472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48kz\" (UniqueName: \"kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.276627 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.276545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.277118 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.277100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.286666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.286640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48kz\" (UniqueName: \"kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz\") pod \"limitador-limitador-8588749977-npslr\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") " pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.400763 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.400729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:03.698094 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.698061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:03.699339 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:44:03.699315    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cf759b_c206_476b_b4d9_b20a9ba936fb.slice/crio-df5f8e4f1c1e5f86a8dc8fd5817eb206a91fcc433d74c950980ba90e7f5926c8 WatchSource:0}: Error finding container df5f8e4f1c1e5f86a8dc8fd5817eb206a91fcc433d74c950980ba90e7f5926c8: Status 404 returned error can't find the container with id df5f8e4f1c1e5f86a8dc8fd5817eb206a91fcc433d74c950980ba90e7f5926c8
Sep 09 18:44:03.904133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:03.904094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-npslr" event={"ID":"45cf759b-c206-476b-b4d9-b20a9ba936fb","Type":"ContainerStarted","Data":"df5f8e4f1c1e5f86a8dc8fd5817eb206a91fcc433d74c950980ba90e7f5926c8"}
Sep 09 18:44:04.100118 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:04.100087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:04.225948 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:04.225906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:06.917647 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:06.917614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-npslr" event={"ID":"45cf759b-c206-476b-b4d9-b20a9ba936fb","Type":"ContainerStarted","Data":"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"}
Sep 09 18:44:06.918064 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:06.917759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:06.942819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:06.942770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-npslr" podStartSLOduration=1.530362095 podStartE2EDuration="3.942755048s" totalImagesPullingTime="2.412392953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:06.942583641 +0000 UTC m=+753.174190002" watchObservedRunningTime="2026-09-09 18:44:06.942755048 +0000 UTC m=+753.174361409"
Sep 09 18:44:17.888233 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:17.888194    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:17.888678 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:17.888419    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-npslr" podUID="45cf759b-c206-476b-b4d9-b20a9ba936fb" containerName="limitador" containerID="cri-o://182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9" gracePeriod=30
Sep 09 18:44:17.889129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:17.889085    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:18.431675 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.431650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:18.494722 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.494623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file\") pod \"45cf759b-c206-476b-b4d9-b20a9ba936fb\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") "
Sep 09 18:44:18.494722 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.494669    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q48kz\" (UniqueName: \"kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz\") pod \"45cf759b-c206-476b-b4d9-b20a9ba936fb\" (UID: \"45cf759b-c206-476b-b4d9-b20a9ba936fb\") "
Sep 09 18:44:18.495007 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.494982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file" (OuterVolumeSpecName: "config-file") pod "45cf759b-c206-476b-b4d9-b20a9ba936fb" (UID: "45cf759b-c206-476b-b4d9-b20a9ba936fb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:44:18.496928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.496909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz" (OuterVolumeSpecName: "kube-api-access-q48kz") pod "45cf759b-c206-476b-b4d9-b20a9ba936fb" (UID: "45cf759b-c206-476b-b4d9-b20a9ba936fb"). InnerVolumeSpecName "kube-api-access-q48kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:18.596053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.596019    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45cf759b-c206-476b-b4d9-b20a9ba936fb-config-file\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:44:18.596053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.596044    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q48kz\" (UniqueName: \"kubernetes.io/projected/45cf759b-c206-476b-b4d9-b20a9ba936fb-kube-api-access-q48kz\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:44:18.965149 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.965114    2581 generic.go:356] "Generic (PLEG): container finished" podID="45cf759b-c206-476b-b4d9-b20a9ba936fb" containerID="182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9" exitCode=0
Sep 09 18:44:18.965605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.965166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-npslr" event={"ID":"45cf759b-c206-476b-b4d9-b20a9ba936fb","Type":"ContainerDied","Data":"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"}
Sep 09 18:44:18.965605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.965168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-npslr"
Sep 09 18:44:18.965605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.965207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-npslr" event={"ID":"45cf759b-c206-476b-b4d9-b20a9ba936fb","Type":"ContainerDied","Data":"df5f8e4f1c1e5f86a8dc8fd5817eb206a91fcc433d74c950980ba90e7f5926c8"}
Sep 09 18:44:18.965605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.965228    2581 scope.go:117] "RemoveContainer" containerID="182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"
Sep 09 18:44:18.973907 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.973894    2581 scope.go:117] "RemoveContainer" containerID="182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"
Sep 09 18:44:18.974202 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:44:18.974183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9\": container with ID starting with 182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9 not found: ID does not exist" containerID="182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"
Sep 09 18:44:18.974251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.974212    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9"} err="failed to get container status \"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9\": rpc error: code = NotFound desc = could not find container \"182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9\": container with ID starting with 182badc60adc5e9c954adffc014fdbc8f53153e0e93a1f3066e9268764bf65d9 not found: ID does not exist"
Sep 09 18:44:18.990001 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.989979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:18.996757 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:18.996737    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-npslr"]
Sep 09 18:44:19.611069 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.611038    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"]
Sep 09 18:44:19.611323 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.611311    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cf759b-c206-476b-b4d9-b20a9ba936fb" containerName="limitador"
Sep 09 18:44:19.611374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.611324    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cf759b-c206-476b-b4d9-b20a9ba936fb" containerName="limitador"
Sep 09 18:44:19.611416 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.611382    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cf759b-c206-476b-b4d9-b20a9ba936fb" containerName="limitador"
Sep 09 18:44:19.614022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.614005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.617320 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.617296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 18:44:19.617451 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.617416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7rbv4\""
Sep 09 18:44:19.618336 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.618318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 18:44:19.618861 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.618847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 18:44:19.626487 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.626469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"]
Sep 09 18:44:19.703975 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.703939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-data\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.704128 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.704028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkb2\" (UniqueName: \"kubernetes.io/projected/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-kube-api-access-5tkb2\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.805217 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.805184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-data\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.805375 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.805247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkb2\" (UniqueName: \"kubernetes.io/projected/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-kube-api-access-5tkb2\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.805628 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.805610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-data\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.818293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.818263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkb2\" (UniqueName: \"kubernetes.io/projected/e7c0cf24-ef74-492e-b12c-2a186a8e11f2-kube-api-access-5tkb2\") pod \"postgres-69fb4594cf-n69sd\" (UID: \"e7c0cf24-ef74-492e-b12c-2a186a8e11f2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:19.924452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:19.924370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:20.205806 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:20.205781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"]
Sep 09 18:44:20.207133 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:44:20.207106    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c0cf24_ef74_492e_b12c_2a186a8e11f2.slice/crio-3cd33481c42b79d8aa51ee99c7c5e7ba8aef1574d81e3f5e728eb15396f82f98 WatchSource:0}: Error finding container 3cd33481c42b79d8aa51ee99c7c5e7ba8aef1574d81e3f5e728eb15396f82f98: Status 404 returned error can't find the container with id 3cd33481c42b79d8aa51ee99c7c5e7ba8aef1574d81e3f5e728eb15396f82f98
Sep 09 18:44:20.283446 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:20.283416    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45cf759b-c206-476b-b4d9-b20a9ba936fb" path="/var/lib/kubelet/pods/45cf759b-c206-476b-b4d9-b20a9ba936fb/volumes"
Sep 09 18:44:20.974657 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:20.974623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:20.978793 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:20.978759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd" event={"ID":"e7c0cf24-ef74-492e-b12c-2a186a8e11f2","Type":"ContainerStarted","Data":"3cd33481c42b79d8aa51ee99c7c5e7ba8aef1574d81e3f5e728eb15396f82f98"}
Sep 09 18:44:21.085789 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:21.085733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:26.003197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:26.003158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd" event={"ID":"e7c0cf24-ef74-492e-b12c-2a186a8e11f2","Type":"ContainerStarted","Data":"1efcc6e4635da8f7414a9b3971597b445fbc30521a042766af723c44bdc283e2"}
Sep 09 18:44:26.003649 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:26.003309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:26.029108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:26.029055    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd" podStartSLOduration=2.269154727 podStartE2EDuration="7.029039977s" totalImagesPullingTime="4.75988525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:26.028258343 +0000 UTC m=+772.259864706" watchObservedRunningTime="2026-09-09 18:44:26.029039977 +0000 UTC m=+772.260646339"
Sep 09 18:44:32.037060 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:32.037030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n69sd"
Sep 09 18:44:32.378115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:32.378073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:32.659374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:32.659290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:39.811523 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:39.811489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:39.961009 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:39.960970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:40.004832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.004790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:40.021660 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.021634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.024806 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.024782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 18:44:40.024926 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.024903    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ns67g\""
Sep 09 18:44:40.025719 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.025692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 18:44:40.027231 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.027214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 18:44:40.050823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.050795    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:40.169026 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.168988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.169236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.169071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.169236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.169091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4r9\" (UniqueName: \"kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.171764 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.171740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:40.270109 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.270077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.270284 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.270115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4r9\" (UniqueName: \"kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.270284 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.270160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.272836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.272810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.272958 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.272876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.281853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.281829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4r9\" (UniqueName: \"kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9\") pod \"maas-controller-6c8dbf5df5-8rldn\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") " pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.330735 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.330678    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 18:44:40.331250 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.331225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:40.332971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.332950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.350895 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.350863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 18:44:40.471636 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.471533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.471830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.471675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.471830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.471743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46fs\" (UniqueName: \"kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.572664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.572626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46fs\" (UniqueName: \"kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.572664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.572663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.572930 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.572779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.575482 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.575451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.575586 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.575451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.581920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.581895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46fs\" (UniqueName: \"kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs\") pod \"maas-controller-bbb54fb6c-vkbst\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") " pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.607467 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.607400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:40.609666 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:44:40.609633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523dae4d_e1be_4f4f_867d_d8fb057625c2.slice/crio-e8ff8793f1a221fdd680a51202fb0bf2224fb6ea98c4a72a629d02be60d88b02 WatchSource:0}: Error finding container e8ff8793f1a221fdd680a51202fb0bf2224fb6ea98c4a72a629d02be60d88b02: Status 404 returned error can't find the container with id e8ff8793f1a221fdd680a51202fb0bf2224fb6ea98c4a72a629d02be60d88b02
Sep 09 18:44:40.656820 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.656768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:40.933202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:40.933170    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 18:44:40.936152 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:44:40.936131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aa3225_b6e9_49d8_b5e6_d15d482b61ff.slice/crio-dbe9a7551c9b3ebc74299ebb77cf923225185d71783c923d6c389a8226f5e25a WatchSource:0}: Error finding container dbe9a7551c9b3ebc74299ebb77cf923225185d71783c923d6c389a8226f5e25a: Status 404 returned error can't find the container with id dbe9a7551c9b3ebc74299ebb77cf923225185d71783c923d6c389a8226f5e25a
Sep 09 18:44:41.034929 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:41.034895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:41.064562 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:41.064529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" event={"ID":"523dae4d-e1be-4f4f-867d-d8fb057625c2","Type":"ContainerStarted","Data":"e8ff8793f1a221fdd680a51202fb0bf2224fb6ea98c4a72a629d02be60d88b02"}
Sep 09 18:44:41.065647 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:41.065620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" event={"ID":"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff","Type":"ContainerStarted","Data":"dbe9a7551c9b3ebc74299ebb77cf923225185d71783c923d6c389a8226f5e25a"}
Sep 09 18:44:43.483187 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:43.483143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:43.633193 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:43.633165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:44.084857 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.084752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" event={"ID":"523dae4d-e1be-4f4f-867d-d8fb057625c2","Type":"ContainerStarted","Data":"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"}
Sep 09 18:44:44.085027 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.084872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:44.086784 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.086746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" event={"ID":"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff","Type":"ContainerStarted","Data":"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"}
Sep 09 18:44:44.086950 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.086874    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:44.159200 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.159148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" podStartSLOduration=2.534503432 podStartE2EDuration="5.159131935s" totalImagesPullingTime="2.624628503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:44.15836587 +0000 UTC m=+790.389972252" watchObservedRunningTime="2026-09-09 18:44:44.159131935 +0000 UTC m=+790.390738297"
Sep 09 18:44:44.249338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:44.249286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" podStartSLOduration=1.852245032 podStartE2EDuration="4.249267373s" totalImagesPullingTime="2.397022341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:44.248593298 +0000 UTC m=+790.480199661" watchObservedRunningTime="2026-09-09 18:44:44.249267373 +0000 UTC m=+790.480873734"
Sep 09 18:44:52.977035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:52.976995    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:53.079421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:53.079374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:44:55.093393 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.093315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:55.093393 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.093391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 18:44:55.170151 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.170115    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:55.170345 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.170306    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" podUID="523dae4d-e1be-4f4f-867d-d8fb057625c2" containerName="manager" containerID="cri-o://2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14" gracePeriod=30
Sep 09 18:44:55.413851 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.413824    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:55.494308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.494284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs\") pod \"523dae4d-e1be-4f4f-867d-d8fb057625c2\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") "
Sep 09 18:44:55.496488 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.496462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "523dae4d-e1be-4f4f-867d-d8fb057625c2" (UID: "523dae4d-e1be-4f4f-867d-d8fb057625c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:44:55.594812 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.594779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj4r9\" (UniqueName: \"kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9\") pod \"523dae4d-e1be-4f4f-867d-d8fb057625c2\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") "
Sep 09 18:44:55.595002 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.594826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert\") pod \"523dae4d-e1be-4f4f-867d-d8fb057625c2\" (UID: \"523dae4d-e1be-4f4f-867d-d8fb057625c2\") "
Sep 09 18:44:55.595002 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.594976    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:44:55.597026 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.596989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "523dae4d-e1be-4f4f-867d-d8fb057625c2" (UID: "523dae4d-e1be-4f4f-867d-d8fb057625c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:44:55.597140 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.597037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9" (OuterVolumeSpecName: "kube-api-access-pj4r9") pod "523dae4d-e1be-4f4f-867d-d8fb057625c2" (UID: "523dae4d-e1be-4f4f-867d-d8fb057625c2"). InnerVolumeSpecName "kube-api-access-pj4r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:55.695647 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.695560    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/523dae4d-e1be-4f4f-867d-d8fb057625c2-webhook-cert\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:44:55.695647 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:55.695589    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4r9\" (UniqueName: \"kubernetes.io/projected/523dae4d-e1be-4f4f-867d-d8fb057625c2-kube-api-access-pj4r9\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:44:56.140502 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.140470    2581 generic.go:356] "Generic (PLEG): container finished" podID="523dae4d-e1be-4f4f-867d-d8fb057625c2" containerID="2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14" exitCode=0
Sep 09 18:44:56.140991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.140515    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn"
Sep 09 18:44:56.140991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.140558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" event={"ID":"523dae4d-e1be-4f4f-867d-d8fb057625c2","Type":"ContainerDied","Data":"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"}
Sep 09 18:44:56.140991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.140614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c8dbf5df5-8rldn" event={"ID":"523dae4d-e1be-4f4f-867d-d8fb057625c2","Type":"ContainerDied","Data":"e8ff8793f1a221fdd680a51202fb0bf2224fb6ea98c4a72a629d02be60d88b02"}
Sep 09 18:44:56.140991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.140648    2581 scope.go:117] "RemoveContainer" containerID="2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"
Sep 09 18:44:56.148908 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.148893    2581 scope.go:117] "RemoveContainer" containerID="2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"
Sep 09 18:44:56.149162 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:44:56.149144    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14\": container with ID starting with 2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14 not found: ID does not exist" containerID="2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"
Sep 09 18:44:56.149210 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.149171    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14"} err="failed to get container status \"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14\": rpc error: code = NotFound desc = could not find container \"2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14\": container with ID starting with 2980799b94a0cedbadc574a33a476e5de4c03111387a9f9a6996599574362d14 not found: ID does not exist"
Sep 09 18:44:56.186182 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.186143    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:56.187527 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.187503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c8dbf5df5-8rldn"]
Sep 09 18:44:56.287601 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:44:56.287571    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523dae4d-e1be-4f4f-867d-d8fb057625c2" path="/var/lib/kubelet/pods/523dae4d-e1be-4f4f-867d-d8fb057625c2/volumes"
Sep 09 18:45:05.408005 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:05.407967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:15.410263 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:15.410227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:15.584749 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:15.584695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:16.754944 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:16.754916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:16.899393 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:16.899352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:17.057519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.057434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:17.070659 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.070607    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 18:45:17.070939 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.070925    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523dae4d-e1be-4f4f-867d-d8fb057625c2" containerName="manager"
Sep 09 18:45:17.071023 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.070941    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="523dae4d-e1be-4f4f-867d-d8fb057625c2" containerName="manager"
Sep 09 18:45:17.071023 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.071005    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="523dae4d-e1be-4f4f-867d-d8fb057625c2" containerName="manager"
Sep 09 18:45:17.078522 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.078505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.084034 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.084012    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wsrn9\""
Sep 09 18:45:17.085149 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.085129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 18:45:17.086302 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.086285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 18:45:17.096886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.096858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 18:45:17.157746 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.157690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.157936 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.157812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.157936 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.157838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt49r\" (UniqueName: \"kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.157936 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.157858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.258775 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.258744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.258947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.258794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.258947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.258816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt49r\" (UniqueName: \"kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.258947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.258850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.259398 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.259377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.261598 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.261571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.261688 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.261571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.268428 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.268404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt49r\" (UniqueName: \"kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r\") pod \"maas-api-5cb87fb765-gmhcj\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") " pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.331981 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.331903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:17.388615 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.388579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:17.565568 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.565536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:17.670390 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.670356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 18:45:17.673399 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:45:17.673373    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1a9018_5d2b_4352_b462_6f9bb50b1d13.slice/crio-1f51fb2d51b5612786faefd2a395fc71a764047a25df00500e7a25d357f27980 WatchSource:0}: Error finding container 1f51fb2d51b5612786faefd2a395fc71a764047a25df00500e7a25d357f27980: Status 404 returned error can't find the container with id 1f51fb2d51b5612786faefd2a395fc71a764047a25df00500e7a25d357f27980
Sep 09 18:45:17.673581 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.673558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:17.693063 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.693039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"]
Sep 09 18:45:17.697717 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.697691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.700693 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.700665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 18:45:17.709664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.709643    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"]
Sep 09 18:45:17.762053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.762014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26rt\" (UniqueName: \"kubernetes.io/projected/7c3a4a98-65ff-4b54-a949-92ba038e691a-kube-api-access-b26rt\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.762454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.762060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3a4a98-65ff-4b54-a949-92ba038e691a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.862548 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.862516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26rt\" (UniqueName: \"kubernetes.io/projected/7c3a4a98-65ff-4b54-a949-92ba038e691a-kube-api-access-b26rt\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.862768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.862559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3a4a98-65ff-4b54-a949-92ba038e691a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.863300 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.863276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3a4a98-65ff-4b54-a949-92ba038e691a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:17.888802 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:17.888775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26rt\" (UniqueName: \"kubernetes.io/projected/7c3a4a98-65ff-4b54-a949-92ba038e691a-kube-api-access-b26rt\") pod \"payload-pre-processing-59fbc47bf8-wfthn\" (UID: \"7c3a4a98-65ff-4b54-a949-92ba038e691a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:18.008400 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:18.008314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:18.228840 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:18.228800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" event={"ID":"6d1a9018-5d2b-4352-b462-6f9bb50b1d13","Type":"ContainerStarted","Data":"1f51fb2d51b5612786faefd2a395fc71a764047a25df00500e7a25d357f27980"}
Sep 09 18:45:18.297796 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:18.297700    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"]
Sep 09 18:45:18.300937 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:45:18.300902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3a4a98_65ff_4b54_a949_92ba038e691a.slice/crio-a7c69e4df1a00def3fab64cf5cbcccca8b774f28adfce85214b61a7942151400 WatchSource:0}: Error finding container a7c69e4df1a00def3fab64cf5cbcccca8b774f28adfce85214b61a7942151400: Status 404 returned error can't find the container with id a7c69e4df1a00def3fab64cf5cbcccca8b774f28adfce85214b61a7942151400
Sep 09 18:45:19.235346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:19.235303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn" event={"ID":"7c3a4a98-65ff-4b54-a949-92ba038e691a","Type":"ContainerStarted","Data":"a7c69e4df1a00def3fab64cf5cbcccca8b774f28adfce85214b61a7942151400"}
Sep 09 18:45:22.254580 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:22.254537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn" event={"ID":"7c3a4a98-65ff-4b54-a949-92ba038e691a","Type":"ContainerStarted","Data":"ce414a7ef0c6523a2a6940ec516430154e146b717a7869ff59a2c1052813707d"}
Sep 09 18:45:22.255025 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:22.254691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:22.292392 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:22.292327    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn" podStartSLOduration=2.081589308 podStartE2EDuration="5.292310133s" totalImagesPullingTime="3.210720825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:22.290486615 +0000 UTC m=+828.522092979" watchObservedRunningTime="2026-09-09 18:45:22.292310133 +0000 UTC m=+828.523916494"
Sep 09 18:45:23.261173 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:23.261136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" event={"ID":"6d1a9018-5d2b-4352-b462-6f9bb50b1d13","Type":"ContainerStarted","Data":"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"}
Sep 09 18:45:23.261606 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:23.261201    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:23.280022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:23.279972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" podStartSLOduration=1.385561343 podStartE2EDuration="6.279956162s" totalImagesPullingTime="4.894394819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:23.278671752 +0000 UTC m=+829.510278116" watchObservedRunningTime="2026-09-09 18:45:23.279956162 +0000 UTC m=+829.511562523"
Sep 09 18:45:29.271621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:29.271594    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 18:45:30.355209 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:30.355174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:30.450911 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:30.450884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:33.004427 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:33.004392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:33.263937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:33.263858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wfthn"
Sep 09 18:45:37.685731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.685668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"]
Sep 09 18:45:37.690189 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.690167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.697129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.697107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 18:45:37.697985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.697969    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 18:45:37.698122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.698111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9cfsg\""
Sep 09 18:45:37.698158 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.698122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 18:45:37.708702 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.708674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"]
Sep 09 18:45:37.731234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.731211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:37.839173 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.839173 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f309db75-d227-447d-bab3-548d1b077896-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.839414 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.839414 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82rl\" (UniqueName: \"kubernetes.io/projected/f309db75-d227-447d-bab3-548d1b077896-kube-api-access-l82rl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.839414 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.839526 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.839430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.939923 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.939832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82rl\" (UniqueName: \"kubernetes.io/projected/f309db75-d227-447d-bab3-548d1b077896-kube-api-access-l82rl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.939923 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.939897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.939941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.939964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.939983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f309db75-d227-447d-bab3-548d1b077896-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.940025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940456 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.940423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940554 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.940439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.940554 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.940485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.942496 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.942473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f309db75-d227-447d-bab3-548d1b077896-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.942625 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.942606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f309db75-d227-447d-bab3-548d1b077896-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:37.951257 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:37.951232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82rl\" (UniqueName: \"kubernetes.io/projected/f309db75-d227-447d-bab3-548d1b077896-kube-api-access-l82rl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dz7vc\" (UID: \"f309db75-d227-447d-bab3-548d1b077896\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:38.000305 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:38.000273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:38.276160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:38.276006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"]
Sep 09 18:45:38.278757 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:45:38.278730    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf309db75_d227_447d_bab3_548d1b077896.slice/crio-989d40b9d87304c1d285606c813a822c1b495522a544f11e211021a3cb538097 WatchSource:0}: Error finding container 989d40b9d87304c1d285606c813a822c1b495522a544f11e211021a3cb538097: Status 404 returned error can't find the container with id 989d40b9d87304c1d285606c813a822c1b495522a544f11e211021a3cb538097
Sep 09 18:45:38.325800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:38.325765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc" event={"ID":"f309db75-d227-447d-bab3-548d1b077896","Type":"ContainerStarted","Data":"989d40b9d87304c1d285606c813a822c1b495522a544f11e211021a3cb538097"}
Sep 09 18:45:40.337165 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:40.337133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc" event={"ID":"f309db75-d227-447d-bab3-548d1b077896","Type":"ContainerStarted","Data":"c9a94c5611e52f4c84a78189d791f11c4baf14febffc751cca5ecdd6c71616f2"}
Sep 09 18:45:40.359183 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:40.359123    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc" podStartSLOduration=2.368958369 podStartE2EDuration="3.359107395s" totalImagesPullingTime="990.149026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:40.357991917 +0000 UTC m=+846.589598305" watchObservedRunningTime="2026-09-09 18:45:40.359107395 +0000 UTC m=+846.590713757"
Sep 09 18:45:48.001167 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.001125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:48.001588 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.001191    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:48.013757 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.013724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:48.379118 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.379091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dz7vc"
Sep 09 18:45:48.651609 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.651528    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"]
Sep 09 18:45:48.655150 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.655127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.657891 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.657872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 18:45:48.664758 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.664735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"]
Sep 09 18:45:48.673095 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.673075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:48.727461 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpnb\" (UniqueName: \"kubernetes.io/projected/20ad08f4-4325-4fa0-a92f-a27d8995cd34-kube-api-access-7vpnb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.727620 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.727620 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.727620 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.727620 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.727804 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.727652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpnb\" (UniqueName: \"kubernetes.io/projected/20ad08f4-4325-4fa0-a92f-a27d8995cd34-kube-api-access-7vpnb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.828978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.830342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.829447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.833785 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.832697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.833785 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.833283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.834211 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.834173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.835882 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.835849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ad08f4-4325-4fa0-a92f-a27d8995cd34-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.841041 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.841021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpnb\" (UniqueName: \"kubernetes.io/projected/20ad08f4-4325-4fa0-a92f-a27d8995cd34-kube-api-access-7vpnb\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk\" (UID: \"20ad08f4-4325-4fa0-a92f-a27d8995cd34\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:48.966651 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:48.966562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:49.237375 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:49.237286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"]
Sep 09 18:45:49.240166 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:45:49.240137    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ad08f4_4325_4fa0_a92f_a27d8995cd34.slice/crio-169b3c05c7d2b2432d486022204f7bdeda86fe5700e429d7f67c7563b490a47a WatchSource:0}: Error finding container 169b3c05c7d2b2432d486022204f7bdeda86fe5700e429d7f67c7563b490a47a: Status 404 returned error can't find the container with id 169b3c05c7d2b2432d486022204f7bdeda86fe5700e429d7f67c7563b490a47a
Sep 09 18:45:49.241504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:49.241489    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:45:49.373599 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:49.373565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk" event={"ID":"20ad08f4-4325-4fa0-a92f-a27d8995cd34","Type":"ContainerStarted","Data":"169b3c05c7d2b2432d486022204f7bdeda86fe5700e429d7f67c7563b490a47a"}
Sep 09 18:45:50.379859 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:50.379817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk" event={"ID":"20ad08f4-4325-4fa0-a92f-a27d8995cd34","Type":"ContainerStarted","Data":"a3acf3486a7f37902d63d1ef4eefae24118f1add48bfcbd18d8a3484ef405a0c"}
Sep 09 18:45:50.397795 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:50.397740    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk" podStartSLOduration=2.200874935 podStartE2EDuration="2.397723961s" totalImagesPullingTime="196.849026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:50.396116543 +0000 UTC m=+856.627722904" watchObservedRunningTime="2026-09-09 18:45:50.397723961 +0000 UTC m=+856.629330327"
Sep 09 18:45:53.854519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:53.854488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:45:58.967332 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:58.967298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:58.967797 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:58.967357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:58.980096 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:58.980071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:45:59.429852 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:45:59.429822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk"
Sep 09 18:46:21.892484 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:21.892433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:46:37.439194 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.439164    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"]
Sep 09 18:46:37.442627 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.442605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.445266 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.445235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 18:46:37.454218 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.454194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"]
Sep 09 18:46:37.466544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.466513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:46:37.532632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.532823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06fc30-622b-4eff-ae24-57e283604c2a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.532823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg89j\" (UniqueName: \"kubernetes.io/projected/2f06fc30-622b-4eff-ae24-57e283604c2a-kube-api-access-lg89j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.532823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.532823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.533037 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.532855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.633758 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.633927 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.633927 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.633927 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.634080 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06fc30-622b-4eff-ae24-57e283604c2a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.634080 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.633995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg89j\" (UniqueName: \"kubernetes.io/projected/2f06fc30-622b-4eff-ae24-57e283604c2a-kube-api-access-lg89j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.634179 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.634132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.634179 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.634157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.634281 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.634199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.636397 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.636367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f06fc30-622b-4eff-ae24-57e283604c2a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.636659 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.636638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06fc30-622b-4eff-ae24-57e283604c2a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.645209 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.645189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg89j\" (UniqueName: \"kubernetes.io/projected/2f06fc30-622b-4eff-ae24-57e283604c2a-kube-api-access-lg89j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d59vw\" (UID: \"2f06fc30-622b-4eff-ae24-57e283604c2a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:37.752643 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:37.752565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:38.025075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:38.024918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"]
Sep 09 18:46:38.027685 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:46:38.027654    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f06fc30_622b_4eff_ae24_57e283604c2a.slice/crio-a95807e6d0548ae17ce4b9ccf2c9fe5ea891c92cd28391ccd6d44d093690e7fa WatchSource:0}: Error finding container a95807e6d0548ae17ce4b9ccf2c9fe5ea891c92cd28391ccd6d44d093690e7fa: Status 404 returned error can't find the container with id a95807e6d0548ae17ce4b9ccf2c9fe5ea891c92cd28391ccd6d44d093690e7fa
Sep 09 18:46:38.583785 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:38.583674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw" event={"ID":"2f06fc30-622b-4eff-ae24-57e283604c2a","Type":"ContainerStarted","Data":"893f2523a328123dd419d1b845801b2e49dabffc847a788a9b36afb480a8cf0c"}
Sep 09 18:46:38.583785 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:38.583752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw" event={"ID":"2f06fc30-622b-4eff-ae24-57e283604c2a","Type":"ContainerStarted","Data":"a95807e6d0548ae17ce4b9ccf2c9fe5ea891c92cd28391ccd6d44d093690e7fa"}
Sep 09 18:46:38.602836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:38.602791    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw" podStartSLOduration=1.393343447 podStartE2EDuration="1.602775949s" totalImagesPullingTime="209.432502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:46:38.602117906 +0000 UTC m=+904.833724322" watchObservedRunningTime="2026-09-09 18:46:38.602775949 +0000 UTC m=+904.834382323"
Sep 09 18:46:47.753016 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:47.752967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:47.753456 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:47.753047    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:47.765594 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:47.765570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:48.638245 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:48.638210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d59vw"
Sep 09 18:46:49.035156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:46:49.035082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:47:52.129454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:47:52.129424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:47:52.225241 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:47:52.225206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:47:53.362231 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:47:53.362200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:47:53.487690 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:47:53.487662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:09.290286 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:09.290206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:09.400661 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:09.400625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:09.631994 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:09.631962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:09.737181 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:09.737127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:10.798399 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:10.798365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:10.901671 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:10.901630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:11.265843 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:11.265812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:13.188029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:13.187981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:13.436026 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:13.435993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:14.329148 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.329118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:48:14.332432 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.332411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.334880 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.334861    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-90f815-serving-cert\""
Sep 09 18:48:14.345019 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.344988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:48:14.375728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.375689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:14.461468 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.461433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlx7\" (UniqueName: \"kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.461655 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.461479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.461655 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.461545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.461655 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.461594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.472945 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.472917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:14.562100 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.562040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.562290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.562135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlx7\" (UniqueName: \"kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.562290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.562174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.562290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.562198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.562784 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.562763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.564723 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.564689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.564824 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.564798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.575161 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.575135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:14.576747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.576725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlx7\" (UniqueName: \"kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7\") pod \"maas-api-e2e-inf-a-90f815-549c5b5568-vps4l\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.642790 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.642749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:14.836847 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.836811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:14.924980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.924955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:48:14.927522 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:14.927493    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f087bda_e64a_4287_aa3e_fd82fe1ebd32.slice/crio-775cab3d29e979c8f84c336ccc635f824b4f3694f1cbc7fd3131ed93cabab2c8 WatchSource:0}: Error finding container 775cab3d29e979c8f84c336ccc635f824b4f3694f1cbc7fd3131ed93cabab2c8: Status 404 returned error can't find the container with id 775cab3d29e979c8f84c336ccc635f824b4f3694f1cbc7fd3131ed93cabab2c8
Sep 09 18:48:14.927660 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:14.927596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:15.003302 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.003266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" event={"ID":"7f087bda-e64a-4287-aa3e-fd82fe1ebd32","Type":"ContainerStarted","Data":"775cab3d29e979c8f84c336ccc635f824b4f3694f1cbc7fd3131ed93cabab2c8"}
Sep 09 18:48:15.025582 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.025544    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:48:15.030035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.030010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.033177 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.033153    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-90f815\""
Sep 09 18:48:15.033326 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.033309    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-90f815-dockercfg-x686q\""
Sep 09 18:48:15.042103 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.037856    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:48:15.168078 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.168028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjds\" (UniqueName: \"kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.168078 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.168077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.268980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.268898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjds\" (UniqueName: \"kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.268980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.268941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.269579 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.269561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.277723 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.277684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjds\" (UniqueName: \"kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds\") pod \"payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.343094 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.343050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:15.615290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:15.615260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:48:15.617657 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:15.617628    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a8b343_0dc9_420d_83af_789a6c3859db.slice/crio-db9c7a2ad13baee33dd6aac2ddfdedb0c927e06153e041ff59cafed0bd5b5367 WatchSource:0}: Error finding container db9c7a2ad13baee33dd6aac2ddfdedb0c927e06153e041ff59cafed0bd5b5367: Status 404 returned error can't find the container with id db9c7a2ad13baee33dd6aac2ddfdedb0c927e06153e041ff59cafed0bd5b5367
Sep 09 18:48:16.010160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.010045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" event={"ID":"17a8b343-0dc9-420d-83af-789a6c3859db","Type":"ContainerStarted","Data":"a21ccd1409f155f034b00c9c9a694e10cb62ed2ffbb252099480f352a6f36318"}
Sep 09 18:48:16.010160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.010109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" event={"ID":"17a8b343-0dc9-420d-83af-789a6c3859db","Type":"ContainerStarted","Data":"db9c7a2ad13baee33dd6aac2ddfdedb0c927e06153e041ff59cafed0bd5b5367"}
Sep 09 18:48:16.010160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.010137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:16.011835 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.011803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" event={"ID":"7f087bda-e64a-4287-aa3e-fd82fe1ebd32","Type":"ContainerStarted","Data":"41e06e4b27777baaafed2f0c905c95a66f779feb6e1f093493bdda185163af79"}
Sep 09 18:48:16.011983 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.011965    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:16.028108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.028062    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" podStartSLOduration=1.028043673 podStartE2EDuration="1.028043673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:16.027128105 +0000 UTC m=+1002.258734478" watchObservedRunningTime="2026-09-09 18:48:16.028043673 +0000 UTC m=+1002.259650036"
Sep 09 18:48:16.047100 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:16.047050    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" podStartSLOduration=1.5718086100000002 podStartE2EDuration="2.047033475s" totalImagesPullingTime="475.224865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:16.045545801 +0000 UTC m=+1002.277152179" watchObservedRunningTime="2026-09-09 18:48:16.047033475 +0000 UTC m=+1002.278639837"
Sep 09 18:48:17.139826 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.139796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:17.245456 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.245425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:17.385322 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.385291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:17.731764 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.731699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:17.792739 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.792644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:17.896946 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:17.896902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:20.592616 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:20.592581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:20.677513 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:20.677485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:21.836346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.836311    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:48:21.840952 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.840928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:21.843882 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.843856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0c2fa5c-serving-cert\""
Sep 09 18:48:21.852019 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.851994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:48:21.898521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.898477    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:21.977255 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:21.977224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:22.023138 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.023112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:48:22.030729 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.030671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.030729 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.030725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.030896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.030770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.030896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.030786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrcd\" (UniqueName: \"kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.132016 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.131976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.132206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.132034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.132206 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:48:22.132121    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-serving-cert: secret "maas-api-e2e-ait-d0c2fa5c-serving-cert" not found
Sep 09 18:48:22.132206 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:48:22.132186    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls podName:be11dd72-4fe3-420f-a822-f73717cd946f nodeName:}" failed. No retries permitted until 2026-09-09 18:48:22.632168922 +0000 UTC m=+1008.863775262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls") pod "maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" (UID: "be11dd72-4fe3-420f-a822-f73717cd946f") : secret "maas-api-e2e-ait-d0c2fa5c-serving-cert" not found
Sep 09 18:48:22.132378 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.132215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.132378 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.132243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrcd\" (UniqueName: \"kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.132953 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.132925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.134667 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.134646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.141420 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.141402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrcd\" (UniqueName: \"kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.372861 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.372821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:22.471661 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.471565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:22.635619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.635589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.638171 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.638143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") pod \"maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:22.753419 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:22.753321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:23.027849 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:23.027822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:48:23.028175 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:23.027846    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe11dd72_4fe3_420f_a822_f73717cd946f.slice/crio-be861eedd36d1a923ea374bc8d31191650c4a7b02dbbc485915af4488db233cf WatchSource:0}: Error finding container be861eedd36d1a923ea374bc8d31191650c4a7b02dbbc485915af4488db233cf: Status 404 returned error can't find the container with id be861eedd36d1a923ea374bc8d31191650c4a7b02dbbc485915af4488db233cf
Sep 09 18:48:23.046384 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:23.046350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" event={"ID":"be11dd72-4fe3-420f-a822-f73717cd946f","Type":"ContainerStarted","Data":"be861eedd36d1a923ea374bc8d31191650c4a7b02dbbc485915af4488db233cf"}
Sep 09 18:48:24.053535 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:24.053433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" event={"ID":"be11dd72-4fe3-420f-a822-f73717cd946f","Type":"ContainerStarted","Data":"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"}
Sep 09 18:48:24.053535 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:24.053521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:27.020303 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:27.020266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:48:27.036539 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:27.036494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" podStartSLOduration=5.293503802 podStartE2EDuration="6.03647787s" totalImagesPullingTime="742.974068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:24.071424665 +0000 UTC m=+1010.303031027" watchObservedRunningTime="2026-09-09 18:48:27.03647787 +0000 UTC m=+1013.268084232"
Sep 09 18:48:28.333975 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:28.333936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:28.489410 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:28.489375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:28.618025 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:28.617977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:28.726419 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:28.726383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:29.600271 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:29.600238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:29.939259 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:29.939185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:30.065009 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:30.064981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:48:30.387143 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:30.387107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:31.446963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:31.446925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:33.317512 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:33.317482    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:33.418425 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:33.418375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:34.449633 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.449601    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:48:34.453004 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.452984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.455807 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.455786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-38af9d-serving-cert\""
Sep 09 18:48:34.463454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.463428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:48:34.535262 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.535225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwm4j\" (UniqueName: \"kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.535262 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.535262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.535490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.535281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.535490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.535301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.628666 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.628622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:34.635795 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.635769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.635928 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.635911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwm4j\" (UniqueName: \"kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.635993 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.635951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.635993 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.635978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.636368 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.636348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.638452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.638430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.638534 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.638490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.652963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.652938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwm4j\" (UniqueName: \"kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j\") pod \"maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:34.734164 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.734072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:34.766385 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:34.766350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:35.010338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.010259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:35.046433 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.046407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:48:35.048126 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:35.048099    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef252a3_1938_4971_9a61_77a03ef89278.slice/crio-ecdbe568c3406d0f3f2d26e288691bec87b21a047d899420b369d59eb203609b WatchSource:0}: Error finding container ecdbe568c3406d0f3f2d26e288691bec87b21a047d899420b369d59eb203609b: Status 404 returned error can't find the container with id ecdbe568c3406d0f3f2d26e288691bec87b21a047d899420b369d59eb203609b
Sep 09 18:48:35.107124 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.107092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:35.109216 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.109193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" event={"ID":"5ef252a3-1938-4971-9a61-77a03ef89278","Type":"ContainerStarted","Data":"ecdbe568c3406d0f3f2d26e288691bec87b21a047d899420b369d59eb203609b"}
Sep 09 18:48:35.115413 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.115380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:48:35.119722 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.119688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.122635 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.122611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-38af9d-dockercfg-88hwz\""
Sep 09 18:48:35.122738 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.122619    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-38af9d\""
Sep 09 18:48:35.134906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.134884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:48:35.140424 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.140401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dp4\" (UniqueName: \"kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.140539 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.140458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.241082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.241045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dp4\" (UniqueName: \"kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.241265 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.241098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.241858 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.241836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.250279 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.250251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dp4\" (UniqueName: \"kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4\") pod \"payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.430160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.430124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:35.570649 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.570617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:35.674933 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.674897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:35.710297 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:35.710259    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96398842_062a_4dfc_a1ea_d298ec436d70.slice/crio-b758c16f87fe9f52015f2f027f2c85b34d516135ded23b2eadc6f8b74edf2062 WatchSource:0}: Error finding container b758c16f87fe9f52015f2f027f2c85b34d516135ded23b2eadc6f8b74edf2062: Status 404 returned error can't find the container with id b758c16f87fe9f52015f2f027f2c85b34d516135ded23b2eadc6f8b74edf2062
Sep 09 18:48:35.711747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:35.711719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:48:36.116293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.116249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" event={"ID":"5ef252a3-1938-4971-9a61-77a03ef89278","Type":"ContainerStarted","Data":"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"}
Sep 09 18:48:36.116783 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.116326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:36.117925 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.117892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" event={"ID":"96398842-062a-4dfc-a1ea-d298ec436d70","Type":"ContainerStarted","Data":"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"}
Sep 09 18:48:36.118017 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.117932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" event={"ID":"96398842-062a-4dfc-a1ea-d298ec436d70","Type":"ContainerStarted","Data":"b758c16f87fe9f52015f2f027f2c85b34d516135ded23b2eadc6f8b74edf2062"}
Sep 09 18:48:36.118070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.118048    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:36.133513 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.133454    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" podStartSLOduration=1.464254583 podStartE2EDuration="2.133438875s" totalImagesPullingTime="669.184292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:36.132459819 +0000 UTC m=+1022.364066203" watchObservedRunningTime="2026-09-09 18:48:36.133438875 +0000 UTC m=+1022.365045236"
Sep 09 18:48:36.149962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.149913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" podStartSLOduration=1.149897745 podStartE2EDuration="1.149897745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:36.147973906 +0000 UTC m=+1022.379580269" watchObservedRunningTime="2026-09-09 18:48:36.149897745 +0000 UTC m=+1022.381504107"
Sep 09 18:48:36.379269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:36.379234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:37.177021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:37.176992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:38.082838 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:38.082809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:38.407157 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:38.407124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:38.732591 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:38.732502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:38.830017 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:38.829985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:41.354996 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:41.354959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:41.451125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:41.451075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:42.128890 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.128861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:48:42.494178 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.494107    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:48:42.498966 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.498944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.501416 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.501389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-77aadf-serving-cert\""
Sep 09 18:48:42.512619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.512594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:48:42.573927 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.573901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:42.610166 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.610125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.610356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.610177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.610356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.610209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.610356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.610281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2s6\" (UniqueName: \"kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.678439 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.678410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:42.711618 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.711589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.711790 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.711626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.711790 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.711676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2s6\" (UniqueName: \"kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.711790 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.711755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.711790 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:48:42.711761    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-serving-cert: secret "maas-api-e2e-inf-b-77aadf-serving-cert" not found
Sep 09 18:48:42.712095 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:48:42.711825    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls podName:12fa65a4-2d92-47e2-8748-978ce52ef7c9 nodeName:}" failed. No retries permitted until 2026-09-09 18:48:43.211807645 +0000 UTC m=+1029.443413999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls") pod "maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" (UID: "12fa65a4-2d92-47e2-8748-978ce52ef7c9") : secret "maas-api-e2e-inf-b-77aadf-serving-cert" not found
Sep 09 18:48:42.712436 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.712418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.714311 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.714289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.724442 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.724413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2s6\" (UniqueName: \"kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:42.838021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:42.837947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:43.180918 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.180889    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:43.216453 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.216417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:43.219090 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.219067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:43.273965 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.273934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:43.410626 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.410579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:43.689546 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.689520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:48:43.691484 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:48:43.691461    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fa65a4_2d92_47e2_8748_978ce52ef7c9.slice/crio-cc91b13e7289658d1f486e69040bc8f6a8d574ed68b71abfce7cdffe1f703030 WatchSource:0}: Error finding container cc91b13e7289658d1f486e69040bc8f6a8d574ed68b71abfce7cdffe1f703030: Status 404 returned error can't find the container with id cc91b13e7289658d1f486e69040bc8f6a8d574ed68b71abfce7cdffe1f703030
Sep 09 18:48:43.819962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:43.819852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:44.155860 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:44.155826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" event={"ID":"12fa65a4-2d92-47e2-8748-978ce52ef7c9","Type":"ContainerStarted","Data":"cc91b13e7289658d1f486e69040bc8f6a8d574ed68b71abfce7cdffe1f703030"}
Sep 09 18:48:44.725296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:44.725260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:45.161876 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.161846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" event={"ID":"12fa65a4-2d92-47e2-8748-978ce52ef7c9","Type":"ContainerStarted","Data":"06cb89b5e08046d1d0d0b7d4ca0f1abac54f3c5a5a3f2e45b85a51a35bde00b6"}
Sep 09 18:48:45.162036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.161907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:45.186390 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.186332    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" podStartSLOduration=2.69281683 podStartE2EDuration="3.186310825s" totalImagesPullingTime="493.493995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:45.183541348 +0000 UTC m=+1031.415147711" watchObservedRunningTime="2026-09-09 18:48:45.186310825 +0000 UTC m=+1031.417917188"
Sep 09 18:48:45.419653 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.419572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:45.554961 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.554926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:45.669997 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.669922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:45.817417 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:45.817387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:47.125392 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:47.125363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:48:51.172615 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:51.172588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:48:53.069250 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:53.069211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:53.172076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:53.172039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:53.269246 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:53.269209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:53.417185 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:53.417160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:55.416346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:55.416317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:58.040319 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:58.040284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:58.417001 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:58.416965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:59.690156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:59.690122    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:48:59.790509 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:48:59.790435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:02.045566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.045514    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:02.555356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.555319    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:49:02.555626 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.555600    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" podUID="be11dd72-4fe3-420f-a822-f73717cd946f" containerName="maas-api" containerID="cri-o://d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee" gracePeriod=30
Sep 09 18:49:02.753125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.753091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:02.810698 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.810633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:49:02.824288 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.824265    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:02.973900 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:02.973854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:03.003939 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.003903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs\") pod \"be11dd72-4fe3-420f-a822-f73717cd946f\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") "
Sep 09 18:49:03.004086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.003972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdrcd\" (UniqueName: \"kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd\") pod \"be11dd72-4fe3-420f-a822-f73717cd946f\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") "
Sep 09 18:49:03.004086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.004016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca\") pod \"be11dd72-4fe3-420f-a822-f73717cd946f\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") "
Sep 09 18:49:03.004086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.004052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") pod \"be11dd72-4fe3-420f-a822-f73717cd946f\" (UID: \"be11dd72-4fe3-420f-a822-f73717cd946f\") "
Sep 09 18:49:03.004445 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.004417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be11dd72-4fe3-420f-a822-f73717cd946f" (UID: "be11dd72-4fe3-420f-a822-f73717cd946f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:03.006354 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.006325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be11dd72-4fe3-420f-a822-f73717cd946f" (UID: "be11dd72-4fe3-420f-a822-f73717cd946f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:49:03.006354 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.006353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be11dd72-4fe3-420f-a822-f73717cd946f" (UID: "be11dd72-4fe3-420f-a822-f73717cd946f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:49:03.006505 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.006370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd" (OuterVolumeSpecName: "kube-api-access-wdrcd") pod "be11dd72-4fe3-420f-a822-f73717cd946f" (UID: "be11dd72-4fe3-420f-a822-f73717cd946f"). InnerVolumeSpecName "kube-api-access-wdrcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:03.104768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.104740    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.104768 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.104768    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrcd\" (UniqueName: \"kubernetes.io/projected/be11dd72-4fe3-420f-a822-f73717cd946f-kube-api-access-wdrcd\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.105160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.104777    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be11dd72-4fe3-420f-a822-f73717cd946f-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.105160 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.104786    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be11dd72-4fe3-420f-a822-f73717cd946f-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.256884 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.256846    2581 generic.go:356] "Generic (PLEG): container finished" podID="be11dd72-4fe3-420f-a822-f73717cd946f" containerID="d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee" exitCode=0
Sep 09 18:49:03.257075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.256900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" event={"ID":"be11dd72-4fe3-420f-a822-f73717cd946f","Type":"ContainerDied","Data":"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"}
Sep 09 18:49:03.257075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.256915    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"
Sep 09 18:49:03.257075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.256975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87" event={"ID":"be11dd72-4fe3-420f-a822-f73717cd946f","Type":"ContainerDied","Data":"be861eedd36d1a923ea374bc8d31191650c4a7b02dbbc485915af4488db233cf"}
Sep 09 18:49:03.257075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.257005    2581 scope.go:117] "RemoveContainer" containerID="d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"
Sep 09 18:49:03.266516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.266498    2581 scope.go:117] "RemoveContainer" containerID="d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"
Sep 09 18:49:03.266804 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:49:03.266786    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee\": container with ID starting with d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee not found: ID does not exist" containerID="d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"
Sep 09 18:49:03.266862 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.266813    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee"} err="failed to get container status \"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee\": rpc error: code = NotFound desc = could not find container \"d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee\": container with ID starting with d35668e3931a5ff5cbccc7176cba748de361c6c67e504ee8a97e24c80976e7ee not found: ID does not exist"
Sep 09 18:49:03.307122 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.306945    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:49:03.317170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.317144    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0c2fa5c-55897dc888-6sk87"]
Sep 09 18:49:03.588366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.588284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:03.681019 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.680987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:03.795533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:03.795502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:04.163822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:04.163791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:04.266985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:04.266945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:04.284375 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:04.284347    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be11dd72-4fe3-420f-a822-f73717cd946f" path="/var/lib/kubelet/pods/be11dd72-4fe3-420f-a822-f73717cd946f/volumes"
Sep 09 18:49:06.445917 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:06.445885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:07.146873 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.146836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:07.350796 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.350760    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:07.351131 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.351116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be11dd72-4fe3-420f-a822-f73717cd946f" containerName="maas-api"
Sep 09 18:49:07.351131 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.351129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be11dd72-4fe3-420f-a822-f73717cd946f" containerName="maas-api"
Sep 09 18:49:07.351285 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.351202    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be11dd72-4fe3-420f-a822-f73717cd946f" containerName="maas-api"
Sep 09 18:49:07.355493 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.355468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.358099 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.358072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b6a21af8-serving-cert\""
Sep 09 18:49:07.365420 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.365400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:07.439429 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.439326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.439429 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.439384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.439621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.439467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.439621 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.439533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5bn\" (UniqueName: \"kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.540187 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.540151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.540630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.540205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.540630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.540234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5bn\" (UniqueName: \"kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.540630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.540287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.540630 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:49:07.540370    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-serving-cert: secret "maas-api-e2e-migrate-b6a21af8-serving-cert" not found
Sep 09 18:49:07.540630 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:49:07.540426    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls podName:6d8942a4-04c9-4fcd-a892-b7c58180e1b8 nodeName:}" failed. No retries permitted until 2026-09-09 18:49:08.040408474 +0000 UTC m=+1054.272014814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls") pod "maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" (UID: "6d8942a4-04c9-4fcd-a892-b7c58180e1b8") : secret "maas-api-e2e-migrate-b6a21af8-serving-cert" not found
Sep 09 18:49:07.540953 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.540930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.542918 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.542882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:07.554382 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:07.554354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5bn\" (UniqueName: \"kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:08.022552 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.022517    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:08.026354 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.026336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.028952 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.028932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b6a21af8-dockercfg-69snr\""
Sep 09 18:49:08.029068 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.028950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b6a21af8\""
Sep 09 18:49:08.039018 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.038991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:08.044370 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.044341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfzb\" (UniqueName: \"kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.044469 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.044444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.044510 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.044495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:08.046913 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.046887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") pod \"maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:08.130408 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.130379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:08.145579 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.145547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.145745 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.145635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfzb\" (UniqueName: \"kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.146260 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.146238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.154269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.154240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfzb\" (UniqueName: \"kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb\") pod \"payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.228219 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.228184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:08.268583 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.268542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:08.339139 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.339107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:08.543900 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.543820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:08.546188 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:49:08.546154    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8942a4_04c9_4fcd_a892_b7c58180e1b8.slice/crio-1a9f7f4fb491ef24f685e8e399186f99bd588f4adb7a9dfa3cea8518b1f4872b WatchSource:0}: Error finding container 1a9f7f4fb491ef24f685e8e399186f99bd588f4adb7a9dfa3cea8518b1f4872b: Status 404 returned error can't find the container with id 1a9f7f4fb491ef24f685e8e399186f99bd588f4adb7a9dfa3cea8518b1f4872b
Sep 09 18:49:08.616255 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:08.616229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:08.618874 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:49:08.618842    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22400a9b_69bc_4dda_bf20_c9494a6a1503.slice/crio-ac017b6d5cccb87a8e43dc88f0a25a96311196810b01318581b15e00922ba9d5 WatchSource:0}: Error finding container ac017b6d5cccb87a8e43dc88f0a25a96311196810b01318581b15e00922ba9d5: Status 404 returned error can't find the container with id ac017b6d5cccb87a8e43dc88f0a25a96311196810b01318581b15e00922ba9d5
Sep 09 18:49:09.288592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.288494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" event={"ID":"22400a9b-69bc-4dda-bf20-c9494a6a1503","Type":"ContainerStarted","Data":"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"}
Sep 09 18:49:09.288592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.288566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" event={"ID":"22400a9b-69bc-4dda-bf20-c9494a6a1503","Type":"ContainerStarted","Data":"ac017b6d5cccb87a8e43dc88f0a25a96311196810b01318581b15e00922ba9d5"}
Sep 09 18:49:09.288842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.288602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:09.290351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.290327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" event={"ID":"6d8942a4-04c9-4fcd-a892-b7c58180e1b8","Type":"ContainerStarted","Data":"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"}
Sep 09 18:49:09.290500 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.290368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" event={"ID":"6d8942a4-04c9-4fcd-a892-b7c58180e1b8","Type":"ContainerStarted","Data":"1a9f7f4fb491ef24f685e8e399186f99bd588f4adb7a9dfa3cea8518b1f4872b"}
Sep 09 18:49:09.290500 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.290389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:09.305173 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.305133    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" podStartSLOduration=1.3051196520000001 podStartE2EDuration="1.305119652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:09.304323788 +0000 UTC m=+1055.535930166" watchObservedRunningTime="2026-09-09 18:49:09.305119652 +0000 UTC m=+1055.536726041"
Sep 09 18:49:09.323591 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:09.323548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" podStartSLOduration=1.946730644 podStartE2EDuration="2.323535276s" totalImagesPullingTime="376.804632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:09.321264498 +0000 UTC m=+1055.552870861" watchObservedRunningTime="2026-09-09 18:49:09.323535276 +0000 UTC m=+1055.555141686"
Sep 09 18:49:14.040028 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:14.039997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:15.300499 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:15.300466    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:20.297094 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:20.297063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:34.576095 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:34.576014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:38.726225 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:38.726194    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:38.726697 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:38.726423    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" podUID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" containerName="maas-api" containerID="cri-o://637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe" gracePeriod=30
Sep 09 18:49:38.761584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:38.761553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:38.798368 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:38.798327    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:38.798604 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:38.798575    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" podUID="22400a9b-69bc-4dda-bf20-c9494a6a1503" containerName="payload-processing" containerID="cri-o://485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686" gracePeriod=30
Sep 09 18:49:39.045980 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.045952    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:39.078977 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.078950    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:39.144731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.139157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca\") pod \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") "
Sep 09 18:49:39.144731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.139276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs\") pod \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") "
Sep 09 18:49:39.144731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.139311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm5bn\" (UniqueName: \"kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn\") pod \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") "
Sep 09 18:49:39.144731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.139346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") pod \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\" (UID: \"6d8942a4-04c9-4fcd-a892-b7c58180e1b8\") "
Sep 09 18:49:39.145504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.145452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d8942a4-04c9-4fcd-a892-b7c58180e1b8" (UID: "6d8942a4-04c9-4fcd-a892-b7c58180e1b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:39.148694 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.148665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d8942a4-04c9-4fcd-a892-b7c58180e1b8" (UID: "6d8942a4-04c9-4fcd-a892-b7c58180e1b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:49:39.149659 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.149632    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn" (OuterVolumeSpecName: "kube-api-access-lm5bn") pod "6d8942a4-04c9-4fcd-a892-b7c58180e1b8" (UID: "6d8942a4-04c9-4fcd-a892-b7c58180e1b8"). InnerVolumeSpecName "kube-api-access-lm5bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:39.151595 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.151568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d8942a4-04c9-4fcd-a892-b7c58180e1b8" (UID: "6d8942a4-04c9-4fcd-a892-b7c58180e1b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:49:39.240384 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfzb\" (UniqueName: \"kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb\") pod \"22400a9b-69bc-4dda-bf20-c9494a6a1503\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") "
Sep 09 18:49:39.240548 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240398    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume\") pod \"22400a9b-69bc-4dda-bf20-c9494a6a1503\" (UID: \"22400a9b-69bc-4dda-bf20-c9494a6a1503\") "
Sep 09 18:49:39.240548 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240538    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.240623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240550    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.240623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240565    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5bn\" (UniqueName: \"kubernetes.io/projected/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-kube-api-access-lm5bn\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.240623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240586    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8942a4-04c9-4fcd-a892-b7c58180e1b8-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.240805 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.240776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22400a9b-69bc-4dda-bf20-c9494a6a1503" (UID: "22400a9b-69bc-4dda-bf20-c9494a6a1503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:39.242615 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.242588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb" (OuterVolumeSpecName: "kube-api-access-vdfzb") pod "22400a9b-69bc-4dda-bf20-c9494a6a1503" (UID: "22400a9b-69bc-4dda-bf20-c9494a6a1503"). InnerVolumeSpecName "kube-api-access-vdfzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:39.341360 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.341261    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfzb\" (UniqueName: \"kubernetes.io/projected/22400a9b-69bc-4dda-bf20-c9494a6a1503-kube-api-access-vdfzb\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.341360 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.341301    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22400a9b-69bc-4dda-bf20-c9494a6a1503-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.436257 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.436217    2581 generic.go:356] "Generic (PLEG): container finished" podID="22400a9b-69bc-4dda-bf20-c9494a6a1503" containerID="485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686" exitCode=0
Sep 09 18:49:39.436411 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.436275    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"
Sep 09 18:49:39.436411 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.436298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" event={"ID":"22400a9b-69bc-4dda-bf20-c9494a6a1503","Type":"ContainerDied","Data":"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"}
Sep 09 18:49:39.436411 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.436387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r" event={"ID":"22400a9b-69bc-4dda-bf20-c9494a6a1503","Type":"ContainerDied","Data":"ac017b6d5cccb87a8e43dc88f0a25a96311196810b01318581b15e00922ba9d5"}
Sep 09 18:49:39.436596 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.436416    2581 scope.go:117] "RemoveContainer" containerID="485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"
Sep 09 18:49:39.437893 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.437866    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" containerID="637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe" exitCode=0
Sep 09 18:49:39.438008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.437918    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"
Sep 09 18:49:39.438008 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.437955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" event={"ID":"6d8942a4-04c9-4fcd-a892-b7c58180e1b8","Type":"ContainerDied","Data":"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"}
Sep 09 18:49:39.438113 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.438009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5" event={"ID":"6d8942a4-04c9-4fcd-a892-b7c58180e1b8","Type":"ContainerDied","Data":"1a9f7f4fb491ef24f685e8e399186f99bd588f4adb7a9dfa3cea8518b1f4872b"}
Sep 09 18:49:39.445744 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.445702    2581 scope.go:117] "RemoveContainer" containerID="485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"
Sep 09 18:49:39.446062 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:49:39.446044    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686\": container with ID starting with 485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686 not found: ID does not exist" containerID="485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"
Sep 09 18:49:39.446145 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.446068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686"} err="failed to get container status \"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686\": rpc error: code = NotFound desc = could not find container \"485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686\": container with ID starting with 485487bd710611be03ed529b9bda812b86e600e9b60126c0c2cc831c30402686 not found: ID does not exist"
Sep 09 18:49:39.446145 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.446084    2581 scope.go:117] "RemoveContainer" containerID="637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"
Sep 09 18:49:39.454288 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.454269    2581 scope.go:117] "RemoveContainer" containerID="637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"
Sep 09 18:49:39.454542 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:49:39.454523    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe\": container with ID starting with 637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe not found: ID does not exist" containerID="637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"
Sep 09 18:49:39.454606 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.454554    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe"} err="failed to get container status \"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe\": rpc error: code = NotFound desc = could not find container \"637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe\": container with ID starting with 637c8d741128d2f03a8d53f384f2045303b531d5f48b14ab710b940de7a58cfe not found: ID does not exist"
Sep 09 18:49:39.458871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.458848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:39.468971 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.468950    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6a21af8-f5bb6d686-lwp7r"]
Sep 09 18:49:39.480369 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.480337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:39.484260 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.484236    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6a21af8-5b7464b456-csds5"]
Sep 09 18:49:39.964295 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:39.964102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:40.097461 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:40.097381    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:40.284462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:40.284427    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22400a9b-69bc-4dda-bf20-c9494a6a1503" path="/var/lib/kubelet/pods/22400a9b-69bc-4dda-bf20-c9494a6a1503/volumes"
Sep 09 18:49:40.284891 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:40.284875    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" path="/var/lib/kubelet/pods/6d8942a4-04c9-4fcd-a892-b7c58180e1b8/volumes"
Sep 09 18:49:42.301243 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:42.301212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:42.407734 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:42.407682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:43.358082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358044    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:49:43.358463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358396    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22400a9b-69bc-4dda-bf20-c9494a6a1503" containerName="payload-processing"
Sep 09 18:49:43.358463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358407    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22400a9b-69bc-4dda-bf20-c9494a6a1503" containerName="payload-processing"
Sep 09 18:49:43.358463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358431    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" containerName="maas-api"
Sep 09 18:49:43.358463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358438    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" containerName="maas-api"
Sep 09 18:49:43.358683 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358492    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8942a4-04c9-4fcd-a892-b7c58180e1b8" containerName="maas-api"
Sep 09 18:49:43.358683 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.358502    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22400a9b-69bc-4dda-bf20-c9494a6a1503" containerName="payload-processing"
Sep 09 18:49:43.361688 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.361667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.365162 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.365141    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90f815\"/\"openshift-service-ca.crt\""
Sep 09 18:49:43.366140 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.366122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90f815\"/\"default-dockercfg-trvbn\""
Sep 09 18:49:43.366251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.366164    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90f815\"/\"kube-root-ca.crt\""
Sep 09 18:49:43.366251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.366182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90f815\"/\"test-model-90f815-kserve-self-signed-certs\""
Sep 09 18:49:43.373979 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.373954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:49:43.459254 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.459206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:43.479512 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.479671 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.479671 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d8z9\" (UniqueName: \"kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.479671 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.479848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.479848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.479753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.559813 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.559772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:49:43.563785 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.563758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.567450 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.567425    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-65f3bfa3-serving-cert\""
Sep 09 18:49:43.581056 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d8z9\" (UniqueName: \"kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581419 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581474 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581474 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.581584 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.581571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.583573 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.583545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.584146 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.584125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.605353 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.598597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:49:43.614614 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.614528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d8z9\" (UniqueName: \"kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9\") pod \"test-model-90f815-kserve-75ddc769bc-qhfsz\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") " pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.639819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.639783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:43.671188 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.671153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:43.682007 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.681973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.682133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.682046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.682133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.682077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7xz\" (UniqueName: \"kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.682133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.682103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.742651 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.742615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:43.783453 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.783413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.783632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.783487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.783632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.783519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7xz\" (UniqueName: \"kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.783632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.783544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.784346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.784318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.786370 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.786341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.786544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.786528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.796625 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.796595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7xz\" (UniqueName: \"kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz\") pod \"maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.834032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.833997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:43.874823 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.874737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:43.970572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:43.970545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:49:43.973177 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:49:43.973137    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf6129f_f87f_44e2_8d37_2115471b45a1.slice/crio-62af1c6a393781e5bcb2cd6f3fd4ad0fbbd5225ea137c5cfce76369d6a690979 WatchSource:0}: Error finding container 62af1c6a393781e5bcb2cd6f3fd4ad0fbbd5225ea137c5cfce76369d6a690979: Status 404 returned error can't find the container with id 62af1c6a393781e5bcb2cd6f3fd4ad0fbbd5225ea137c5cfce76369d6a690979
Sep 09 18:49:44.162499 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.162474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:49:44.164387 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:49:44.164363    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285d7e63_c724_4122_ab0e_75f873fc38fd.slice/crio-0a66c4ce01f0d7130b76e8ae4dfedea3aa2e12d1f7a83d1a15597c950ae07d21 WatchSource:0}: Error finding container 0a66c4ce01f0d7130b76e8ae4dfedea3aa2e12d1f7a83d1a15597c950ae07d21: Status 404 returned error can't find the container with id 0a66c4ce01f0d7130b76e8ae4dfedea3aa2e12d1f7a83d1a15597c950ae07d21
Sep 09 18:49:44.316821 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.316792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:44.416014 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.415938    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:44.467470 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.467432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" event={"ID":"5cf6129f-f87f-44e2-8d37-2115471b45a1","Type":"ContainerStarted","Data":"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"}
Sep 09 18:49:44.467669 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.467479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" event={"ID":"5cf6129f-f87f-44e2-8d37-2115471b45a1","Type":"ContainerStarted","Data":"62af1c6a393781e5bcb2cd6f3fd4ad0fbbd5225ea137c5cfce76369d6a690979"}
Sep 09 18:49:44.468592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.468562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" event={"ID":"285d7e63-c724-4122-ab0e-75f873fc38fd","Type":"ContainerStarted","Data":"0a66c4ce01f0d7130b76e8ae4dfedea3aa2e12d1f7a83d1a15597c950ae07d21"}
Sep 09 18:49:44.487565 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:44.487505    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" podStartSLOduration=1.487485951 podStartE2EDuration="1.487485951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:44.484413415 +0000 UTC m=+1090.716019778" watchObservedRunningTime="2026-09-09 18:49:44.487485951 +0000 UTC m=+1090.719092313"
Sep 09 18:49:45.475137 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:45.475102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" event={"ID":"285d7e63-c724-4122-ab0e-75f873fc38fd","Type":"ContainerStarted","Data":"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"}
Sep 09 18:49:45.475570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:45.475281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:45.504338 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:45.504287    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" podStartSLOduration=1.892816898 podStartE2EDuration="2.50427299s" totalImagesPullingTime="611.456092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:45.501661675 +0000 UTC m=+1091.733268038" watchObservedRunningTime="2026-09-09 18:49:45.50427299 +0000 UTC m=+1091.735879392"
Sep 09 18:49:46.950634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:46.950601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:51.486140 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:51.486109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:49:53.671993 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:53.671958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:53.672357 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:53.672011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:53.685379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:53.685343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:49:53.858605 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:53.858577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:54.198341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:54.198308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:49:54.528260 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:49:54.528178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:50:04.625589 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.625555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:50:04.626010 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.625792    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" podUID="5ef252a3-1938-4971-9a61-77a03ef89278" containerName="maas-api" containerID="cri-o://cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5" gracePeriod=30
Sep 09 18:50:04.668241 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.668209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:04.703625 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.703596    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:50:04.703860 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.703828    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" podUID="96398842-062a-4dfc-a1ea-d298ec436d70" containerName="payload-pre-processing" containerID="cri-o://52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd" gracePeriod=30
Sep 09 18:50:04.981966 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.981939    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:50:04.985242 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:04.985221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:50:05.072167 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls\") pod \"5ef252a3-1938-4971-9a61-77a03ef89278\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") "
Sep 09 18:50:05.072351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca\") pod \"5ef252a3-1938-4971-9a61-77a03ef89278\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") "
Sep 09 18:50:05.072351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwm4j\" (UniqueName: \"kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j\") pod \"5ef252a3-1938-4971-9a61-77a03ef89278\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") "
Sep 09 18:50:05.072351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs\") pod \"5ef252a3-1938-4971-9a61-77a03ef89278\" (UID: \"5ef252a3-1938-4971-9a61-77a03ef89278\") "
Sep 09 18:50:05.072351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66dp4\" (UniqueName: \"kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4\") pod \"96398842-062a-4dfc-a1ea-d298ec436d70\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") "
Sep 09 18:50:05.072351 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume\") pod \"96398842-062a-4dfc-a1ea-d298ec436d70\" (UID: \"96398842-062a-4dfc-a1ea-d298ec436d70\") "
Sep 09 18:50:05.072733 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ef252a3-1938-4971-9a61-77a03ef89278" (UID: "5ef252a3-1938-4971-9a61-77a03ef89278"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:05.072919 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.072814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96398842-062a-4dfc-a1ea-d298ec436d70" (UID: "96398842-062a-4dfc-a1ea-d298ec436d70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:05.074528 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.074501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ef252a3-1938-4971-9a61-77a03ef89278" (UID: "5ef252a3-1938-4971-9a61-77a03ef89278"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:05.074624 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.074529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4" (OuterVolumeSpecName: "kube-api-access-66dp4") pod "96398842-062a-4dfc-a1ea-d298ec436d70" (UID: "96398842-062a-4dfc-a1ea-d298ec436d70"). InnerVolumeSpecName "kube-api-access-66dp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:05.075075 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.075058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ef252a3-1938-4971-9a61-77a03ef89278" (UID: "5ef252a3-1938-4971-9a61-77a03ef89278"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:05.075148 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.075069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j" (OuterVolumeSpecName: "kube-api-access-wwm4j") pod "5ef252a3-1938-4971-9a61-77a03ef89278" (UID: "5ef252a3-1938-4971-9a61-77a03ef89278"). InnerVolumeSpecName "kube-api-access-wwm4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:05.173583 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173547    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwm4j\" (UniqueName: \"kubernetes.io/projected/5ef252a3-1938-4971-9a61-77a03ef89278-kube-api-access-wwm4j\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.173583 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173580    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.173822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173594    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dp4\" (UniqueName: \"kubernetes.io/projected/96398842-062a-4dfc-a1ea-d298ec436d70-kube-api-access-66dp4\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.173822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173606    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96398842-062a-4dfc-a1ea-d298ec436d70-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.173822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173617    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef252a3-1938-4971-9a61-77a03ef89278-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.173822 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.173629    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef252a3-1938-4971-9a61-77a03ef89278-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.385270 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.385243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:05.515513 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.515475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:05.567237 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.567205    2581 generic.go:356] "Generic (PLEG): container finished" podID="96398842-062a-4dfc-a1ea-d298ec436d70" containerID="52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd" exitCode=0
Sep 09 18:50:05.567421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.567247    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"
Sep 09 18:50:05.567421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.567292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" event={"ID":"96398842-062a-4dfc-a1ea-d298ec436d70","Type":"ContainerDied","Data":"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"}
Sep 09 18:50:05.567421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.567347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8" event={"ID":"96398842-062a-4dfc-a1ea-d298ec436d70","Type":"ContainerDied","Data":"b758c16f87fe9f52015f2f027f2c85b34d516135ded23b2eadc6f8b74edf2062"}
Sep 09 18:50:05.567421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.567372    2581 scope.go:117] "RemoveContainer" containerID="52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"
Sep 09 18:50:05.568725 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.568681    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ef252a3-1938-4971-9a61-77a03ef89278" containerID="cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5" exitCode=0
Sep 09 18:50:05.568842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.568728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" event={"ID":"5ef252a3-1938-4971-9a61-77a03ef89278","Type":"ContainerDied","Data":"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"}
Sep 09 18:50:05.568842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.568751    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"
Sep 09 18:50:05.568842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.568781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx" event={"ID":"5ef252a3-1938-4971-9a61-77a03ef89278","Type":"ContainerDied","Data":"ecdbe568c3406d0f3f2d26e288691bec87b21a047d899420b369d59eb203609b"}
Sep 09 18:50:05.576477 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.576431    2581 scope.go:117] "RemoveContainer" containerID="52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"
Sep 09 18:50:05.576751 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:05.576724    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd\": container with ID starting with 52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd not found: ID does not exist" containerID="52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"
Sep 09 18:50:05.576862 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.576760    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd"} err="failed to get container status \"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd\": rpc error: code = NotFound desc = could not find container \"52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd\": container with ID starting with 52524bc944388ebbf67dd711d149cb3f6e1c1af6b5e2927c6e8a77ccf60e28bd not found: ID does not exist"
Sep 09 18:50:05.576862 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.576781    2581 scope.go:117] "RemoveContainer" containerID="cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"
Sep 09 18:50:05.585834 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.585816    2581 scope.go:117] "RemoveContainer" containerID="cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"
Sep 09 18:50:05.586101 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:05.586084    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5\": container with ID starting with cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5 not found: ID does not exist" containerID="cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"
Sep 09 18:50:05.586141 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.586107    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5"} err="failed to get container status \"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5\": rpc error: code = NotFound desc = could not find container \"cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5\": container with ID starting with cbed59ca42234fc439fb3d83ab602f488e2adc5fbee6ca873acc997cf0bb72b5 not found: ID does not exist"
Sep 09 18:50:05.603026 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.603002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:50:05.619582 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.619559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-38af9d-58665bc95d-hj8b8"]
Sep 09 18:50:05.639024 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.638963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:50:05.648265 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:05.648244    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-38af9d-8dd5f4d7d-mfcgx"]
Sep 09 18:50:06.293388 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:06.293351    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef252a3-1938-4971-9a61-77a03ef89278" path="/var/lib/kubelet/pods/5ef252a3-1938-4971-9a61-77a03ef89278/volumes"
Sep 09 18:50:06.293810 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:06.293790    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96398842-062a-4dfc-a1ea-d298ec436d70" path="/var/lib/kubelet/pods/96398842-062a-4dfc-a1ea-d298ec436d70/volumes"
Sep 09 18:50:06.552187 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:06.552097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:12.022221 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:12.022179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:12.151820 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:12.151785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:12.381879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:12.381844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:12.785345 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:12.785257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:12.885235 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:12.885205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:13.127356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:13.127329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:13.197622 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:13.197593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:14.240588 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.240556    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:50:14.241012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.240831    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" podUID="285d7e63-c724-4122-ab0e-75f873fc38fd" containerName="maas-api" containerID="cri-o://2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b" gracePeriod=30
Sep 09 18:50:14.290194 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.290148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:14.373969 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.373930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:14.501818 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.501738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:50:14.614033 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.613997    2581 generic.go:356] "Generic (PLEG): container finished" podID="285d7e63-c724-4122-ab0e-75f873fc38fd" containerID="2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b" exitCode=0
Sep 09 18:50:14.614208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.614051    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"
Sep 09 18:50:14.614208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.614071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" event={"ID":"285d7e63-c724-4122-ab0e-75f873fc38fd","Type":"ContainerDied","Data":"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"}
Sep 09 18:50:14.614208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.614124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz" event={"ID":"285d7e63-c724-4122-ab0e-75f873fc38fd","Type":"ContainerDied","Data":"0a66c4ce01f0d7130b76e8ae4dfedea3aa2e12d1f7a83d1a15597c950ae07d21"}
Sep 09 18:50:14.614208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.614154    2581 scope.go:117] "RemoveContainer" containerID="2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"
Sep 09 18:50:14.622486 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.622467    2581 scope.go:117] "RemoveContainer" containerID="2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"
Sep 09 18:50:14.622804 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:14.622785    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b\": container with ID starting with 2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b not found: ID does not exist" containerID="2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"
Sep 09 18:50:14.622856 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.622814    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b"} err="failed to get container status \"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b\": rpc error: code = NotFound desc = could not find container \"2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b\": container with ID starting with 2aba443b482b2199f91bc7b06e832ba9aef9f74678115ed519a57af96739d52b not found: ID does not exist"
Sep 09 18:50:14.662065 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.662031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp7xz\" (UniqueName: \"kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz\") pod \"285d7e63-c724-4122-ab0e-75f873fc38fd\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") "
Sep 09 18:50:14.662238 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.662119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs\") pod \"285d7e63-c724-4122-ab0e-75f873fc38fd\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") "
Sep 09 18:50:14.662238 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.662182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca\") pod \"285d7e63-c724-4122-ab0e-75f873fc38fd\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") "
Sep 09 18:50:14.662238 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.662204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls\") pod \"285d7e63-c724-4122-ab0e-75f873fc38fd\" (UID: \"285d7e63-c724-4122-ab0e-75f873fc38fd\") "
Sep 09 18:50:14.662608 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.662572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "285d7e63-c724-4122-ab0e-75f873fc38fd" (UID: "285d7e63-c724-4122-ab0e-75f873fc38fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:14.664337 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.664305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz" (OuterVolumeSpecName: "kube-api-access-zp7xz") pod "285d7e63-c724-4122-ab0e-75f873fc38fd" (UID: "285d7e63-c724-4122-ab0e-75f873fc38fd"). InnerVolumeSpecName "kube-api-access-zp7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:14.664830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.664801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "285d7e63-c724-4122-ab0e-75f873fc38fd" (UID: "285d7e63-c724-4122-ab0e-75f873fc38fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:14.664830 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.664821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "285d7e63-c724-4122-ab0e-75f873fc38fd" (UID: "285d7e63-c724-4122-ab0e-75f873fc38fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:14.763269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.763175    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d7e63-c724-4122-ab0e-75f873fc38fd-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.763269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.763209    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.763269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.763220    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7xz\" (UniqueName: \"kubernetes.io/projected/285d7e63-c724-4122-ab0e-75f873fc38fd-kube-api-access-zp7xz\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.763269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.763229    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285d7e63-c724-4122-ab0e-75f873fc38fd-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.929525 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.929502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:14.977301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.977269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:50:14.980914 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:14.980885    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65f3bfa3-6cf5c59d9d-jsdjz"]
Sep 09 18:50:15.055145 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:15.055059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:15.812940 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:15.812901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:16.236158 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.236131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:16.283838 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.283807    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285d7e63-c724-4122-ab0e-75f873fc38fd" path="/var/lib/kubelet/pods/285d7e63-c724-4122-ab0e-75f873fc38fd/volumes"
Sep 09 18:50:16.373901 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.373864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:16.509151 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.509055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:16.803074 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.802998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:16.966256 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:16.966210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:17.505452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505422    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:50:17.505800 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505787    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96398842-062a-4dfc-a1ea-d298ec436d70" containerName="payload-pre-processing"
Sep 09 18:50:17.505869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96398842-062a-4dfc-a1ea-d298ec436d70" containerName="payload-pre-processing"
Sep 09 18:50:17.505869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505823    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d7e63-c724-4122-ab0e-75f873fc38fd" containerName="maas-api"
Sep 09 18:50:17.505869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d7e63-c724-4122-ab0e-75f873fc38fd" containerName="maas-api"
Sep 09 18:50:17.505869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505838    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef252a3-1938-4971-9a61-77a03ef89278" containerName="maas-api"
Sep 09 18:50:17.505869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef252a3-1938-4971-9a61-77a03ef89278" containerName="maas-api"
Sep 09 18:50:17.506063 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505891    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d7e63-c724-4122-ab0e-75f873fc38fd" containerName="maas-api"
Sep 09 18:50:17.506063 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505900    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96398842-062a-4dfc-a1ea-d298ec436d70" containerName="payload-pre-processing"
Sep 09 18:50:17.506063 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.505908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef252a3-1938-4971-9a61-77a03ef89278" containerName="maas-api"
Sep 09 18:50:17.510354 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.510331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.512815 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.512792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f8eac9-serving-cert\""
Sep 09 18:50:17.526021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.525994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:50:17.563112 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.563049    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:17.590021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.589991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.590206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.590050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4klr\" (UniqueName: \"kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.590206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.590134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.590206 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.590158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.664871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.664841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:17.691040 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.690998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.691232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.691068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4klr\" (UniqueName: \"kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.691232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.691112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.691232 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.691134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.691232 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:17.691161    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-serving-cert: secret "maas-api-e2e-explicit-f8eac9-serving-cert" not found
Sep 09 18:50:17.691444 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:17.691239    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls podName:d735c301-8231-4db2-aea4-c2cb6720d3bf nodeName:}" failed. No retries permitted until 2026-09-09 18:50:18.191218887 +0000 UTC m=+1124.422825232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls") pod "maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" (UID: "d735c301-8231-4db2-aea4-c2cb6720d3bf") : secret "maas-api-e2e-explicit-f8eac9-serving-cert" not found
Sep 09 18:50:17.691862 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.691841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.693788 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.693763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.703886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.703864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4klr\" (UniqueName: \"kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:17.861386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:17.861357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:18.196220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.196124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:18.198759 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.198697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") pod \"maas-api-e2e-explicit-f8eac9-698899b8d-mvl29\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:18.303424 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.300512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:18.421439 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.421406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:18.447943 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.447855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:18.698758 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:18.698645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:50:18.701930 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:50:18.701900    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd735c301_8231_4db2_aea4_c2cb6720d3bf.slice/crio-7c25a1a216362444380d0cfe23a67bc342fda683a01dd2d4e69ae226fffc68ea WatchSource:0}: Error finding container 7c25a1a216362444380d0cfe23a67bc342fda683a01dd2d4e69ae226fffc68ea: Status 404 returned error can't find the container with id 7c25a1a216362444380d0cfe23a67bc342fda683a01dd2d4e69ae226fffc68ea
Sep 09 18:50:19.641472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:19.641388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" event={"ID":"d735c301-8231-4db2-aea4-c2cb6720d3bf","Type":"ContainerStarted","Data":"2719d79a3abed559aceb4db941977533352e8528fe7016436928ffc0b945021c"}
Sep 09 18:50:19.641472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:19.641439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" event={"ID":"d735c301-8231-4db2-aea4-c2cb6720d3bf","Type":"ContainerStarted","Data":"7c25a1a216362444380d0cfe23a67bc342fda683a01dd2d4e69ae226fffc68ea"}
Sep 09 18:50:19.641903 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:19.641514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:19.663753 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:19.663685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" podStartSLOduration=2.00462424 podStartE2EDuration="2.663669004s" totalImagesPullingTime="659.044764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:19.660689756 +0000 UTC m=+1125.892296118" watchObservedRunningTime="2026-09-09 18:50:19.663669004 +0000 UTC m=+1125.895275379"
Sep 09 18:50:21.138188 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.138157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:21.141585 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.141563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.144090 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.144065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-edd73ecc-serving-cert\""
Sep 09 18:50:21.153177 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.153152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:21.323646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.323605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.323858 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.323655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.323858 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.323740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.323858 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.323856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxgq\" (UniqueName: \"kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.383108 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.383066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:21.424894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.424805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.424894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.424844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.425111 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.424926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxgq\" (UniqueName: \"kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.425111 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.424984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.425681 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.425653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.427288 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.427262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.427506 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.427485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.434348 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.434302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxgq\" (UniqueName: \"kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq\") pod \"maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.453767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.453732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:21.489333 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.489295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:21.740072 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.740049    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:21.742317 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:50:21.742290    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27cfa2_b393_4803_9b9b_f50728f8b6e4.slice/crio-c1fcabe2cd894d0dc9fca687b8c3ebf8fd0a262b2b9ae92c32c65f5929a65437 WatchSource:0}: Error finding container c1fcabe2cd894d0dc9fca687b8c3ebf8fd0a262b2b9ae92c32c65f5929a65437: Status 404 returned error can't find the container with id c1fcabe2cd894d0dc9fca687b8c3ebf8fd0a262b2b9ae92c32c65f5929a65437
Sep 09 18:50:21.844576 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.844535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:21.940923 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:21.940895    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:22.657529 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:22.657495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" event={"ID":"5e27cfa2-b393-4803-9b9b-f50728f8b6e4","Type":"ContainerStarted","Data":"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"}
Sep 09 18:50:22.658012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:22.657548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" event={"ID":"5e27cfa2-b393-4803-9b9b-f50728f8b6e4","Type":"ContainerStarted","Data":"c1fcabe2cd894d0dc9fca687b8c3ebf8fd0a262b2b9ae92c32c65f5929a65437"}
Sep 09 18:50:22.658012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:22.657644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:22.679356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:22.679303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" podStartSLOduration=1.293188641 podStartE2EDuration="1.679285641s" totalImagesPullingTime="386.097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:22.677207688 +0000 UTC m=+1128.908814094" watchObservedRunningTime="2026-09-09 18:50:22.679285641 +0000 UTC m=+1128.910892003"
Sep 09 18:50:25.652283 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:25.652254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:50:28.667273 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:28.667247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:32.666495 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:32.666460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:32.732483 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:32.732429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:35.075021 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:35.074989    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:39.493921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:39.493883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:40.244887 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:40.244853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:42.473227 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:42.473191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:46.624606 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:46.624539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:51.782333 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:51.782278    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:51.782922 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:51.782596    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" podUID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" containerName="maas-api" containerID="cri-o://a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561" gracePeriod=30
Sep 09 18:50:51.812600 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:51.812575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:51.926510 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:51.926471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:52.039628 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.039566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:52.044306 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.044281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:52.085045 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.085017    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs\") pod \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") "
Sep 09 18:50:52.085226 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.085079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpxgq\" (UniqueName: \"kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq\") pod \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") "
Sep 09 18:50:52.085226 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.085181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca\") pod \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") "
Sep 09 18:50:52.085344 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.085227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls\") pod \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\" (UID: \"5e27cfa2-b393-4803-9b9b-f50728f8b6e4\") "
Sep 09 18:50:52.085658 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.085618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e27cfa2-b393-4803-9b9b-f50728f8b6e4" (UID: "5e27cfa2-b393-4803-9b9b-f50728f8b6e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:52.087405 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.087376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq" (OuterVolumeSpecName: "kube-api-access-vpxgq") pod "5e27cfa2-b393-4803-9b9b-f50728f8b6e4" (UID: "5e27cfa2-b393-4803-9b9b-f50728f8b6e4"). InnerVolumeSpecName "kube-api-access-vpxgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:52.087738 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.087674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e27cfa2-b393-4803-9b9b-f50728f8b6e4" (UID: "5e27cfa2-b393-4803-9b9b-f50728f8b6e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:52.088096 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.088069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e27cfa2-b393-4803-9b9b-f50728f8b6e4" (UID: "5e27cfa2-b393-4803-9b9b-f50728f8b6e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:52.155199 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.155170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:52.186676 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.186643    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.186676 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.186677    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxgq\" (UniqueName: \"kubernetes.io/projected/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-kube-api-access-vpxgq\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.186871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.186695    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.186871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.186745    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e27cfa2-b393-4803-9b9b-f50728f8b6e4-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.288264 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.288222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:52.461081 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.461044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:52.796572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.796540    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" containerID="a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561" exitCode=0
Sep 09 18:50:52.797070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.796584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"
Sep 09 18:50:52.797070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.796628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" event={"ID":"5e27cfa2-b393-4803-9b9b-f50728f8b6e4","Type":"ContainerDied","Data":"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"}
Sep 09 18:50:52.797070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.796690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn" event={"ID":"5e27cfa2-b393-4803-9b9b-f50728f8b6e4","Type":"ContainerDied","Data":"c1fcabe2cd894d0dc9fca687b8c3ebf8fd0a262b2b9ae92c32c65f5929a65437"}
Sep 09 18:50:52.797070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.796749    2581 scope.go:117] "RemoveContainer" containerID="a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"
Sep 09 18:50:52.805481 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.805461    2581 scope.go:117] "RemoveContainer" containerID="a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"
Sep 09 18:50:52.805775 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:50:52.805747    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561\": container with ID starting with a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561 not found: ID does not exist" containerID="a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"
Sep 09 18:50:52.805870 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.805783    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561"} err="failed to get container status \"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561\": rpc error: code = NotFound desc = could not find container \"a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561\": container with ID starting with a1b315fe639b1d8e2abecbcdaccd8cf2d512fc2680a79b4f9b21d22d97ace561 not found: ID does not exist"
Sep 09 18:50:52.819441 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.819417    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:52.824019 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:52.823995    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-edd73ecc-7f6bc95cf7-hnrvn"]
Sep 09 18:50:53.176346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:53.176308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:53.320344 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:53.320313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:50:54.284721 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:54.284670    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" path="/var/lib/kubelet/pods/5e27cfa2-b393-4803-9b9b-f50728f8b6e4/volumes"
Sep 09 18:50:55.063015 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:50:55.062930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:02.541941 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:02.541904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:02.727937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:02.727902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:02.844636 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:02.844550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:03.331959 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:03.331930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:07.349126 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:07.349091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:10.067138 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:10.067107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:10.283734 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:10.283685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:20.682234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.682202    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:51:20.682660 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.682434    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" podUID="d735c301-8231-4db2-aea4-c2cb6720d3bf" containerName="maas-api" containerID="cri-o://2719d79a3abed559aceb4db941977533352e8528fe7016436928ffc0b945021c" gracePeriod=30
Sep 09 18:51:20.714252 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.714222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:20.815534 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.815507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:20.895433 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.895408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:20.933441 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.933355    2581 generic.go:356] "Generic (PLEG): container finished" podID="d735c301-8231-4db2-aea4-c2cb6720d3bf" containerID="2719d79a3abed559aceb4db941977533352e8528fe7016436928ffc0b945021c" exitCode=0
Sep 09 18:51:20.933613 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.933452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" event={"ID":"d735c301-8231-4db2-aea4-c2cb6720d3bf","Type":"ContainerDied","Data":"2719d79a3abed559aceb4db941977533352e8528fe7016436928ffc0b945021c"}
Sep 09 18:51:20.959809 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:20.959787    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:51:21.044873 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.044841    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca\") pod \"d735c301-8231-4db2-aea4-c2cb6720d3bf\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") "
Sep 09 18:51:21.045035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.044917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs\") pod \"d735c301-8231-4db2-aea4-c2cb6720d3bf\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") "
Sep 09 18:51:21.045035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.044976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4klr\" (UniqueName: \"kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr\") pod \"d735c301-8231-4db2-aea4-c2cb6720d3bf\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") "
Sep 09 18:51:21.045127 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.045040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") pod \"d735c301-8231-4db2-aea4-c2cb6720d3bf\" (UID: \"d735c301-8231-4db2-aea4-c2cb6720d3bf\") "
Sep 09 18:51:21.045254 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.045233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d735c301-8231-4db2-aea4-c2cb6720d3bf" (UID: "d735c301-8231-4db2-aea4-c2cb6720d3bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:51:21.045318 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.045298    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d735c301-8231-4db2-aea4-c2cb6720d3bf-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.047212 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.047184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d735c301-8231-4db2-aea4-c2cb6720d3bf" (UID: "d735c301-8231-4db2-aea4-c2cb6720d3bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:51:21.047336 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.047228    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr" (OuterVolumeSpecName: "kube-api-access-m4klr") pod "d735c301-8231-4db2-aea4-c2cb6720d3bf" (UID: "d735c301-8231-4db2-aea4-c2cb6720d3bf"). InnerVolumeSpecName "kube-api-access-m4klr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:51:21.047336 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.047272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d735c301-8231-4db2-aea4-c2cb6720d3bf" (UID: "d735c301-8231-4db2-aea4-c2cb6720d3bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:51:21.075833 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.075800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:21.146791 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.146757    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.146791 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.146784    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4klr\" (UniqueName: \"kubernetes.io/projected/d735c301-8231-4db2-aea4-c2cb6720d3bf-kube-api-access-m4klr\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.146791 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.146794    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d735c301-8231-4db2-aea4-c2cb6720d3bf-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.188056 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.187935    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:21.386129 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.386087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:21.940564 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.940525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29" event={"ID":"d735c301-8231-4db2-aea4-c2cb6720d3bf","Type":"ContainerDied","Data":"7c25a1a216362444380d0cfe23a67bc342fda683a01dd2d4e69ae226fffc68ea"}
Sep 09 18:51:21.940990 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.940612    2581 scope.go:117] "RemoveContainer" containerID="2719d79a3abed559aceb4db941977533352e8528fe7016436928ffc0b945021c"
Sep 09 18:51:21.940990 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.940534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"
Sep 09 18:51:21.976265 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.976226    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:51:21.979350 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:21.979326    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f8eac9-698899b8d-mvl29"]
Sep 09 18:51:22.283833 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:22.283755    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d735c301-8231-4db2-aea4-c2cb6720d3bf" path="/var/lib/kubelet/pods/d735c301-8231-4db2-aea4-c2cb6720d3bf/volumes"
Sep 09 18:51:26.458512 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:26.458483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:26.557982 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:26.557952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:28.289337 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:28.289308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:28.420856 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:28.420826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:28.614047 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:28.614020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:29.016594 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:29.016501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:29.392453 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:29.392420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:29.502844 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:29.502813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:30.086550 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:30.086520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:31.078400 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:31.078368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:31.210289 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:31.210259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:31.343167 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:31.343081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:37.099949 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:37.099918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:37.210892 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:37.210853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:42.284225 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:42.284187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:44.335738 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:44.335677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:51:44.359010 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:44.358981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:52.557261 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:52.557226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:56.486545 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:56.486511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:58.651426 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:58.651392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:59.284435 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.284392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:59.355697 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.355650    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" podUID="5cf6129f-f87f-44e2-8d37-2115471b45a1" containerName="main" containerID="cri-o://ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146" gracePeriod=45
Sep 09 18:51:59.385645 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.385620    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:51:59.605091 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.605068    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:51:59.672808 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.672767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.672808 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.672815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.673236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.673236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:59.673236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d8z9\" (UniqueName: \"kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.673236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.673236 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home\") pod \"5cf6129f-f87f-44e2-8d37-2115471b45a1\" (UID: \"5cf6129f-f87f-44e2-8d37-2115471b45a1\") "
Sep 09 18:51:59.673490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673327    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:59.673490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home" (OuterVolumeSpecName: "home") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:59.673490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673424    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-model-cache\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.673490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.673441    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-tmp-dir\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.675220 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.675199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm" (OuterVolumeSpecName: "dshm") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:59.675306 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.675262    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:51:59.675377 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.675359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9" (OuterVolumeSpecName: "kube-api-access-7d8z9") pod "5cf6129f-f87f-44e2-8d37-2115471b45a1" (UID: "5cf6129f-f87f-44e2-8d37-2115471b45a1"). InnerVolumeSpecName "kube-api-access-7d8z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:51:59.774208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.774160    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-dshm\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.774208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.774201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d8z9\" (UniqueName: \"kubernetes.io/projected/5cf6129f-f87f-44e2-8d37-2115471b45a1-kube-api-access-7d8z9\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.774208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.774212    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cf6129f-f87f-44e2-8d37-2115471b45a1-home\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.774208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.774223    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cf6129f-f87f-44e2-8d37-2115471b45a1-tls-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:51:59.825274 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:51:59.825242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:00.115448 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.115412    2581 generic.go:356] "Generic (PLEG): container finished" podID="5cf6129f-f87f-44e2-8d37-2115471b45a1" containerID="ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146" exitCode=0
Sep 09 18:52:00.115448 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.115454    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"
Sep 09 18:52:00.115732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.115499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" event={"ID":"5cf6129f-f87f-44e2-8d37-2115471b45a1","Type":"ContainerDied","Data":"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"}
Sep 09 18:52:00.115732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.115539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz" event={"ID":"5cf6129f-f87f-44e2-8d37-2115471b45a1","Type":"ContainerDied","Data":"62af1c6a393781e5bcb2cd6f3fd4ad0fbbd5225ea137c5cfce76369d6a690979"}
Sep 09 18:52:00.115732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.115561    2581 scope.go:117] "RemoveContainer" containerID="ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"
Sep 09 18:52:00.124447 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.124432    2581 scope.go:117] "RemoveContainer" containerID="ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"
Sep 09 18:52:00.124761 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:00.124743    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146\": container with ID starting with ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146 not found: ID does not exist" containerID="ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"
Sep 09 18:52:00.124848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.124767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146"} err="failed to get container status \"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146\": rpc error: code = NotFound desc = could not find container \"ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146\": container with ID starting with ba1757a0ad3ef713ca4eb228ee089d5b44e0e171288740634f6f318d16d5c146 not found: ID does not exist"
Sep 09 18:52:00.143933 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.143910    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:52:00.152572 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.152552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-90f815/test-model-90f815-kserve-75ddc769bc-qhfsz"]
Sep 09 18:52:00.289358 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:00.289325    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf6129f-f87f-44e2-8d37-2115471b45a1" path="/var/lib/kubelet/pods/5cf6129f-f87f-44e2-8d37-2115471b45a1/volumes"
Sep 09 18:52:03.922736 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:03.922681    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:52:03.923202 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:03.922959    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" podUID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" containerName="maas-api" containerID="cri-o://41e06e4b27777baaafed2f0c905c95a66f779feb6e1f093493bdda185163af79" gracePeriod=30
Sep 09 18:52:03.964770 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:03.964689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:04.006655 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.006613    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:52:04.006871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.006855    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" podUID="17a8b343-0dc9-420d-83af-789a6c3859db" containerName="payload-pre-processing" containerID="cri-o://a21ccd1409f155f034b00c9c9a694e10cb62ed2ffbb252099480f352a6f36318" gracePeriod=30
Sep 09 18:52:04.078396 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.078366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:04.138452 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.138413    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" containerID="41e06e4b27777baaafed2f0c905c95a66f779feb6e1f093493bdda185163af79" exitCode=0
Sep 09 18:52:04.138641 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.138491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" event={"ID":"7f087bda-e64a-4287-aa3e-fd82fe1ebd32","Type":"ContainerDied","Data":"41e06e4b27777baaafed2f0c905c95a66f779feb6e1f093493bdda185163af79"}
Sep 09 18:52:04.141156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.141090    2581 generic.go:356] "Generic (PLEG): container finished" podID="17a8b343-0dc9-420d-83af-789a6c3859db" containerID="a21ccd1409f155f034b00c9c9a694e10cb62ed2ffbb252099480f352a6f36318" exitCode=0
Sep 09 18:52:04.141156 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.141145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" event={"ID":"17a8b343-0dc9-420d-83af-789a6c3859db","Type":"ContainerDied","Data":"a21ccd1409f155f034b00c9c9a694e10cb62ed2ffbb252099480f352a6f36318"}
Sep 09 18:52:04.184321 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.184249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:04.259010 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.258987    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:52:04.281508 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.281476    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:04.281687 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.281629    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:52:04.316540 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.316512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vlx7\" (UniqueName: \"kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7\") pod \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") "
Sep 09 18:52:04.316728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.316580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs\") pod \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") "
Sep 09 18:52:04.316728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.316644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls\") pod \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") "
Sep 09 18:52:04.316728 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.316671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca\") pod \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\" (UID: \"7f087bda-e64a-4287-aa3e-fd82fe1ebd32\") "
Sep 09 18:52:04.317147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.317116    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f087bda-e64a-4287-aa3e-fd82fe1ebd32" (UID: "7f087bda-e64a-4287-aa3e-fd82fe1ebd32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:04.318864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.318837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f087bda-e64a-4287-aa3e-fd82fe1ebd32" (UID: "7f087bda-e64a-4287-aa3e-fd82fe1ebd32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:04.318864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.318857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f087bda-e64a-4287-aa3e-fd82fe1ebd32" (UID: "7f087bda-e64a-4287-aa3e-fd82fe1ebd32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:04.319116 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.319096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7" (OuterVolumeSpecName: "kube-api-access-4vlx7") pod "7f087bda-e64a-4287-aa3e-fd82fe1ebd32" (UID: "7f087bda-e64a-4287-aa3e-fd82fe1ebd32"). InnerVolumeSpecName "kube-api-access-4vlx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:04.418064 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume\") pod \"17a8b343-0dc9-420d-83af-789a6c3859db\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") "
Sep 09 18:52:04.418234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418145    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssjds\" (UniqueName: \"kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds\") pod \"17a8b343-0dc9-420d-83af-789a6c3859db\" (UID: \"17a8b343-0dc9-420d-83af-789a6c3859db\") "
Sep 09 18:52:04.418340 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418322    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.418379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418349    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.418379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418365    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlx7\" (UniqueName: \"kubernetes.io/projected/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-kube-api-access-4vlx7\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.418444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418379    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f087bda-e64a-4287-aa3e-fd82fe1ebd32-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.418444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.418401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17a8b343-0dc9-420d-83af-789a6c3859db" (UID: "17a8b343-0dc9-420d-83af-789a6c3859db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:04.420402 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.420382    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds" (OuterVolumeSpecName: "kube-api-access-ssjds") pod "17a8b343-0dc9-420d-83af-789a6c3859db" (UID: "17a8b343-0dc9-420d-83af-789a6c3859db"). InnerVolumeSpecName "kube-api-access-ssjds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:04.518956 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.518870    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjds\" (UniqueName: \"kubernetes.io/projected/17a8b343-0dc9-420d-83af-789a6c3859db-kube-api-access-ssjds\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.518956 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.518897    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a8b343-0dc9-420d-83af-789a6c3859db-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.643362 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:04.643333    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:05.146873 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.146837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"
Sep 09 18:52:05.146873 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.146859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn" event={"ID":"17a8b343-0dc9-420d-83af-789a6c3859db","Type":"ContainerDied","Data":"db9c7a2ad13baee33dd6aac2ddfdedb0c927e06153e041ff59cafed0bd5b5367"}
Sep 09 18:52:05.147530 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.146928    2581 scope.go:117] "RemoveContainer" containerID="a21ccd1409f155f034b00c9c9a694e10cb62ed2ffbb252099480f352a6f36318"
Sep 09 18:52:05.148542 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.148189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l" event={"ID":"7f087bda-e64a-4287-aa3e-fd82fe1ebd32","Type":"ContainerDied","Data":"775cab3d29e979c8f84c336ccc635f824b4f3694f1cbc7fd3131ed93cabab2c8"}
Sep 09 18:52:05.148542 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.148196    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"
Sep 09 18:52:05.155947 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.155929    2581 scope.go:117] "RemoveContainer" containerID="41e06e4b27777baaafed2f0c905c95a66f779feb6e1f093493bdda185163af79"
Sep 09 18:52:05.174421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.174398    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:52:05.182171 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.182136    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f815-549c5b5568-vps4l"]
Sep 09 18:52:05.195920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.195869    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:52:05.198739 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:05.198699    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-90f815-6b946b49f8-crjkn"]
Sep 09 18:52:06.283725 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:06.283678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a8b343-0dc9-420d-83af-789a6c3859db" path="/var/lib/kubelet/pods/17a8b343-0dc9-420d-83af-789a6c3859db/volumes"
Sep 09 18:52:06.284080 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:06.284055    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" path="/var/lib/kubelet/pods/7f087bda-e64a-4287-aa3e-fd82fe1ebd32/volumes"
Sep 09 18:52:09.361864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:09.361830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:10.175036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:10.175002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:10.721132 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:10.721102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:10.823328 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:10.823294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:11.057225 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:11.057143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:11.824937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:11.824911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:16.491673 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:16.491639    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:16.578044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:16.578007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:16.672090 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:16.672057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:16.791863 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:16.791777    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:17.630509 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:17.630465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:17.731101 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:17.731069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:17.832839 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:17.832808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:20.176432 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:20.176405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:22.010145 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.010105    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:52:22.011613 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.011552    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" podUID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" containerName="maas-api" containerID="cri-o://06cb89b5e08046d1d0d0b7d4ca0f1abac54f3c5a5a3f2e45b85a51a35bde00b6" gracePeriod=30
Sep 09 18:52:22.039036 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.039009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:22.231193 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.231160    2581 generic.go:356] "Generic (PLEG): container finished" podID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" containerID="06cb89b5e08046d1d0d0b7d4ca0f1abac54f3c5a5a3f2e45b85a51a35bde00b6" exitCode=0
Sep 09 18:52:22.231383 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.231220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" event={"ID":"12fa65a4-2d92-47e2-8748-978ce52ef7c9","Type":"ContainerDied","Data":"06cb89b5e08046d1d0d0b7d4ca0f1abac54f3c5a5a3f2e45b85a51a35bde00b6"}
Sep 09 18:52:22.257990 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.257969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:52:22.374192 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.374156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn2s6\" (UniqueName: \"kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6\") pod \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") "
Sep 09 18:52:22.374384 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.374204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") pod \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") "
Sep 09 18:52:22.374384 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.374233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca\") pod \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") "
Sep 09 18:52:22.374384 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.374298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs\") pod \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\" (UID: \"12fa65a4-2d92-47e2-8748-978ce52ef7c9\") "
Sep 09 18:52:22.374656 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.374623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12fa65a4-2d92-47e2-8748-978ce52ef7c9" (UID: "12fa65a4-2d92-47e2-8748-978ce52ef7c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:22.376416 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.376378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6" (OuterVolumeSpecName: "kube-api-access-jn2s6") pod "12fa65a4-2d92-47e2-8748-978ce52ef7c9" (UID: "12fa65a4-2d92-47e2-8748-978ce52ef7c9"). InnerVolumeSpecName "kube-api-access-jn2s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:22.376416 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.376401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12fa65a4-2d92-47e2-8748-978ce52ef7c9" (UID: "12fa65a4-2d92-47e2-8748-978ce52ef7c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:22.376566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.376462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12fa65a4-2d92-47e2-8748-978ce52ef7c9" (UID: "12fa65a4-2d92-47e2-8748-978ce52ef7c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:22.414312 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.414284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:22.475346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.475312    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2s6\" (UniqueName: \"kubernetes.io/projected/12fa65a4-2d92-47e2-8748-978ce52ef7c9-kube-api-access-jn2s6\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.475346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.475340    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.475346 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.475349    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa65a4-2d92-47e2-8748-978ce52ef7c9-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.475563 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.475357    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fa65a4-2d92-47e2-8748-978ce52ef7c9-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.514623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.514594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:22.621802 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:22.621769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:23.238115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:23.238079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql" event={"ID":"12fa65a4-2d92-47e2-8748-978ce52ef7c9","Type":"ContainerDied","Data":"cc91b13e7289658d1f486e69040bc8f6a8d574ed68b71abfce7cdffe1f703030"}
Sep 09 18:52:23.238115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:23.238095    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"
Sep 09 18:52:23.238561 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:23.238141    2581 scope.go:117] "RemoveContainer" containerID="06cb89b5e08046d1d0d0b7d4ca0f1abac54f3c5a5a3f2e45b85a51a35bde00b6"
Sep 09 18:52:23.261298 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:23.261267    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:52:23.270779 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:23.270753    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77aadf-6b8657d596-p2zql"]
Sep 09 18:52:24.284818 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:24.284782    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" path="/var/lib/kubelet/pods/12fa65a4-2d92-47e2-8748-978ce52ef7c9/volumes"
Sep 09 18:52:25.536603 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:25.536525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:25.637214 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:25.637174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:26.305038 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:26.305007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:28.771146 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:28.771117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:28.868892 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:28.868851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:29.435871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:29.435827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:29.757630 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:29.757537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:30.147123 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:30.147087    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:30.241955 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:30.241901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:32.315601 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:32.315561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:32.416056 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:32.416015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:32.619593 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:32.619564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:32.927903 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:32.927828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:33.273597 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:33.273520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:33.450886 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:33.450850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:34.377011 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.376984    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 18:52:34.377298 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377287    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a8b343-0dc9-420d-83af-789a6c3859db" containerName="payload-pre-processing"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377300    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a8b343-0dc9-420d-83af-789a6c3859db" containerName="payload-pre-processing"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377308    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf6129f-f87f-44e2-8d37-2115471b45a1" containerName="main"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377313    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf6129f-f87f-44e2-8d37-2115471b45a1" containerName="main"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377331    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" containerName="maas-api"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" containerName="maas-api"
Sep 09 18:52:34.377342 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377348    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377357    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d735c301-8231-4db2-aea4-c2cb6720d3bf" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d735c301-8231-4db2-aea4-c2cb6720d3bf" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377369    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377374    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f087bda-e64a-4287-aa3e-fd82fe1ebd32" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377447    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e27cfa2-b393-4803-9b9b-f50728f8b6e4" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377452    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12fa65a4-2d92-47e2-8748-978ce52ef7c9" containerName="maas-api"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377459    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a8b343-0dc9-420d-83af-789a6c3859db" containerName="payload-pre-processing"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377464    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf6129f-f87f-44e2-8d37-2115471b45a1" containerName="main"
Sep 09 18:52:34.377519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.377469    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d735c301-8231-4db2-aea4-c2cb6720d3bf" containerName="maas-api"
Sep 09 18:52:34.381424 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.381409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.383949 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.383929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-282de3-serving-cert\""
Sep 09 18:52:34.395058 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.395029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 18:52:34.422658 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.422631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:34.470491 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.470458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.470963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.470502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.470963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.470596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49wb\" (UniqueName: \"kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.470963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.470647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.538476 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.538445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:34.571921 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.571890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.572086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.571934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.572086 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:34.572045    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-serving-cert: secret "maas-api-e2e-shared-a-w3-282de3-serving-cert" not found
Sep 09 18:52:34.572086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.572068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.572191 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:34.572122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls podName:1c681c76-476f-4d4f-b222-0633d6b3c75d nodeName:}" failed. No retries permitted until 2026-09-09 18:52:35.072099857 +0000 UTC m=+1261.303706196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" (UID: "1c681c76-476f-4d4f-b222-0633d6b3c75d") : secret "maas-api-e2e-shared-a-w3-282de3-serving-cert" not found
Sep 09 18:52:34.572237 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.572200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49wb\" (UniqueName: \"kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.572523 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.572508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.574522 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.574505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:34.582819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:34.582756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49wb\" (UniqueName: \"kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:35.076691 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.076648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:35.079309 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.079276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:35.226749 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.226695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:35.292505 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.292436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:35.574198 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.572313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 18:52:35.575661 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:52:35.575628    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c681c76_476f_4d4f_b222_0633d6b3c75d.slice/crio-841346f576610a528125265ef7475f1708eafeddf551c9bd906cab3dda70a50d WatchSource:0}: Error finding container 841346f576610a528125265ef7475f1708eafeddf551c9bd906cab3dda70a50d: Status 404 returned error can't find the container with id 841346f576610a528125265ef7475f1708eafeddf551c9bd906cab3dda70a50d
Sep 09 18:52:35.577251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.577232    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:52:35.754836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:35.754802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:36.241436 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.241396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:36.300872 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.300793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" event={"ID":"1c681c76-476f-4d4f-b222-0633d6b3c75d","Type":"ContainerStarted","Data":"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"}
Sep 09 18:52:36.300872 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.300841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" event={"ID":"1c681c76-476f-4d4f-b222-0633d6b3c75d","Type":"ContainerStarted","Data":"841346f576610a528125265ef7475f1708eafeddf551c9bd906cab3dda70a50d"}
Sep 09 18:52:36.301047 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.300874    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:36.324282 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.324231    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" podStartSLOduration=1.855605529 podStartE2EDuration="2.32421669s" totalImagesPullingTime="468.611161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:36.323377198 +0000 UTC m=+1262.554983560" watchObservedRunningTime="2026-09-09 18:52:36.32421669 +0000 UTC m=+1262.555823052"
Sep 09 18:52:36.337204 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:36.337177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:37.354386 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.354354    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:52:37.358380 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.358357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.360749 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.360654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ddaae1a6-serving-cert\""
Sep 09 18:52:37.366734 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.366693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:52:37.397916 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.397882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjhm\" (UniqueName: \"kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.398081 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.398046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.398135 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.398106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.398193 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.398171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.400372 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.400351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:37.499698 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.499655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjhm\" (UniqueName: \"kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.499896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.499774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.499896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.499821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.499896 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.499875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.500042 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:37.499947    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-serving-cert: secret "maas-api-e2e-mt-ddaae1a6-serving-cert" not found
Sep 09 18:52:37.500042 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:37.500017    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls podName:cbd7b42e-bc60-44d5-9359-21ae284f51db nodeName:}" failed. No retries permitted until 2026-09-09 18:52:37.999997614 +0000 UTC m=+1264.231603955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls") pod "maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db") : secret "maas-api-e2e-mt-ddaae1a6-serving-cert" not found
Sep 09 18:52:37.500500 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.500477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.502963 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.502941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.516336 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.516312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:37.520494 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.520472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjhm\" (UniqueName: \"kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:37.798435 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.798355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:37.894871 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:37.894839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:38.003430 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.003394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:38.003641 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:38.003582    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-serving-cert: secret "maas-api-e2e-mt-ddaae1a6-serving-cert" not found
Sep 09 18:52:38.003729 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:52:38.003665    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls podName:cbd7b42e-bc60-44d5-9359-21ae284f51db nodeName:}" failed. No retries permitted until 2026-09-09 18:52:39.003642591 +0000 UTC m=+1265.235248956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls") pod "maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db") : secret "maas-api-e2e-mt-ddaae1a6-serving-cert" not found
Sep 09 18:52:38.191983 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.191947    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:52:38.199421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.199385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.202051 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.202031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ddaae1a6-dockercfg-dvjkg\""
Sep 09 18:52:38.204139 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.204122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ddaae1a6\""
Sep 09 18:52:38.213403 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.213381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:52:38.296643 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.296612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:38.305002 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.304970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhnx\" (UniqueName: \"kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.305192 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.305026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.368044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.367994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:38.406127 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.406079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhnx\" (UniqueName: \"kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.406304 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.406159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.406854 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.406834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.415249 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.415217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhnx\" (UniqueName: \"kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx\") pod \"payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") " pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.509529 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.509435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:38.784125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:38.784041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:52:38.788156 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:52:38.788129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f61d33_1d44_4f5e_b88c_290ebb242cad.slice/crio-56e2dded2691e8814179f3c4083850613042e1b115bf20e272e51408538854ff WatchSource:0}: Error finding container 56e2dded2691e8814179f3c4083850613042e1b115bf20e272e51408538854ff: Status 404 returned error can't find the container with id 56e2dded2691e8814179f3c4083850613042e1b115bf20e272e51408538854ff
Sep 09 18:52:39.011308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.011281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:39.014027 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.014000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:39.171531 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.171497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:39.315240 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.315208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" event={"ID":"22f61d33-1d44-4f5e-b88c-290ebb242cad","Type":"ContainerStarted","Data":"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"}
Sep 09 18:52:39.315454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.315261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" event={"ID":"22f61d33-1d44-4f5e-b88c-290ebb242cad","Type":"ContainerStarted","Data":"56e2dded2691e8814179f3c4083850613042e1b115bf20e272e51408538854ff"}
Sep 09 18:52:39.315454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.315282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:39.338252 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.338193    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" podStartSLOduration=1.338176522 podStartE2EDuration="1.338176522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:39.336785156 +0000 UTC m=+1265.568391543" watchObservedRunningTime="2026-09-09 18:52:39.338176522 +0000 UTC m=+1265.569782884"
Sep 09 18:52:39.449285 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:39.449190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:52:39.453146 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:52:39.453116    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd7b42e_bc60_44d5_9359_21ae284f51db.slice/crio-9d3884d92cf8f4fee07a44e6a8de852078a8c4243e56d8b427e0153b4c75d3d9 WatchSource:0}: Error finding container 9d3884d92cf8f4fee07a44e6a8de852078a8c4243e56d8b427e0153b4c75d3d9: Status 404 returned error can't find the container with id 9d3884d92cf8f4fee07a44e6a8de852078a8c4243e56d8b427e0153b4c75d3d9
Sep 09 18:52:40.320634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:40.320592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" event={"ID":"cbd7b42e-bc60-44d5-9359-21ae284f51db","Type":"ContainerStarted","Data":"9d3884d92cf8f4fee07a44e6a8de852078a8c4243e56d8b427e0153b4c75d3d9"}
Sep 09 18:52:41.326774 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:41.326727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" event={"ID":"cbd7b42e-bc60-44d5-9359-21ae284f51db","Type":"ContainerStarted","Data":"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"}
Sep 09 18:52:41.327163 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:41.326892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:41.346601 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:41.346548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" podStartSLOduration=3.361179806 podStartE2EDuration="4.346531996s" totalImagesPullingTime="985.35219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:41.345231257 +0000 UTC m=+1267.576837620" watchObservedRunningTime="2026-09-09 18:52:41.346531996 +0000 UTC m=+1267.578138396"
Sep 09 18:52:42.311193 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:42.311166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 18:52:47.337393 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:47.337364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:52:48.157208 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:48.157175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:48.258046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:48.258009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:50.323157 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:50.323127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:52:55.378977 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:55.378944    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:58.699885 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:58.699835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:58.799413 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:58.799382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:59.111038 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:59.110999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:52:59.334842 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:52:59.334809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:01.814505 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:01.814468    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:02.085988 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:02.085890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:02.186962 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:02.186930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:02.450381 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:02.450348    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:02.602524 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:02.602487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:02.960279 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:02.960247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:03.548224 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.548183    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 18:53:03.556880 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.556854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.559516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.559483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0e012d-serving-cert\""
Sep 09 18:53:03.568510 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.568479    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 18:53:03.603317 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.603281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:03.617691 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.617654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.617906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.617695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.617906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.617810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.617906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.617871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sk5\" (UniqueName: \"kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.674422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.674368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:03.719356 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.719307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.719566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.719366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.719566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.719401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.719566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.719449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sk5\" (UniqueName: \"kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.720076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.720046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.722035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.722014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.722207 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.722187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.732519 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.732487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sk5\" (UniqueName: \"kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5\") pod \"maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:03.771646 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.771605    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:03.868664 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:03.868630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:04.147777 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:04.147665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 18:53:04.151775 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:53:04.151753    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4586543d_6d8e_44fd_a13a_d94a6bbb539f.slice/crio-196822f06b1961230b6965189e768a451516d4dd92ad631902bd15f769d14464 WatchSource:0}: Error finding container 196822f06b1961230b6965189e768a451516d4dd92ad631902bd15f769d14464: Status 404 returned error can't find the container with id 196822f06b1961230b6965189e768a451516d4dd92ad631902bd15f769d14464
Sep 09 18:53:04.435382 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:04.435294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" event={"ID":"4586543d-6d8e-44fd-a13a-d94a6bbb539f","Type":"ContainerStarted","Data":"196822f06b1961230b6965189e768a451516d4dd92ad631902bd15f769d14464"}
Sep 09 18:53:04.594839 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:04.594810    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:04.694639 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:04.694520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:05.441617 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:05.441579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" event={"ID":"4586543d-6d8e-44fd-a13a-d94a6bbb539f","Type":"ContainerStarted","Data":"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"}
Sep 09 18:53:05.442125 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:05.441649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:05.458944 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:05.458893    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" podStartSLOduration=2.027725697 podStartE2EDuration="2.458878113s" totalImagesPullingTime="431.152416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:05.458050326 +0000 UTC m=+1291.689656691" watchObservedRunningTime="2026-09-09 18:53:05.458878113 +0000 UTC m=+1291.690484474"
Sep 09 18:53:08.585152 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:08.585118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:08.687029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:08.686976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:09.260364 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:09.260320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:11.453161 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:11.453134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 18:53:19.626610 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.626573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:53:19.627086 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.626859    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" podUID="cbd7b42e-bc60-44d5-9359-21ae284f51db" containerName="maas-api" containerID="cri-o://f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b" gracePeriod=30
Sep 09 18:53:19.664558 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.664532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:19.699362 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.699316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:53:19.699759 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.699579    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" podUID="22f61d33-1d44-4f5e-b88c-290ebb242cad" containerName="payload-processing" containerID="cri-o://57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9" gracePeriod=30
Sep 09 18:53:19.765314 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.765282    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:19.947246 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.947222    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:53:19.976897 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:19.976875    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:53:20.054716 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.054685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:20.060659 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") pod \"cbd7b42e-bc60-44d5-9359-21ae284f51db\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") "
Sep 09 18:53:20.060794 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs\") pod \"cbd7b42e-bc60-44d5-9359-21ae284f51db\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") "
Sep 09 18:53:20.060856 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fhnx\" (UniqueName: \"kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx\") pod \"22f61d33-1d44-4f5e-b88c-290ebb242cad\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") "
Sep 09 18:53:20.060912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume\") pod \"22f61d33-1d44-4f5e-b88c-290ebb242cad\" (UID: \"22f61d33-1d44-4f5e-b88c-290ebb242cad\") "
Sep 09 18:53:20.060973 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca\") pod \"cbd7b42e-bc60-44d5-9359-21ae284f51db\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") "
Sep 09 18:53:20.060973 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.060954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpjhm\" (UniqueName: \"kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm\") pod \"cbd7b42e-bc60-44d5-9359-21ae284f51db\" (UID: \"cbd7b42e-bc60-44d5-9359-21ae284f51db\") "
Sep 09 18:53:20.061269 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.061235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22f61d33-1d44-4f5e-b88c-290ebb242cad" (UID: "22f61d33-1d44-4f5e-b88c-290ebb242cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:53:20.061375 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.061283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbd7b42e-bc60-44d5-9359-21ae284f51db" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:53:20.063076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.062954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbd7b42e-bc60-44d5-9359-21ae284f51db" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:53:20.063076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.063048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cbd7b42e-bc60-44d5-9359-21ae284f51db" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:53:20.063076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.063055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx" (OuterVolumeSpecName: "kube-api-access-9fhnx") pod "22f61d33-1d44-4f5e-b88c-290ebb242cad" (UID: "22f61d33-1d44-4f5e-b88c-290ebb242cad"). InnerVolumeSpecName "kube-api-access-9fhnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:53:20.063234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.063077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm" (OuterVolumeSpecName: "kube-api-access-kpjhm") pod "cbd7b42e-bc60-44d5-9359-21ae284f51db" (UID: "cbd7b42e-bc60-44d5-9359-21ae284f51db"). InnerVolumeSpecName "kube-api-access-kpjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:53:20.135136 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.135107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161769    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjhm\" (UniqueName: \"kubernetes.io/projected/cbd7b42e-bc60-44d5-9359-21ae284f51db-kube-api-access-kpjhm\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161790    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161801    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd7b42e-bc60-44d5-9359-21ae284f51db-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161809    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fhnx\" (UniqueName: \"kubernetes.io/projected/22f61d33-1d44-4f5e-b88c-290ebb242cad-kube-api-access-9fhnx\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161819    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f61d33-1d44-4f5e-b88c-290ebb242cad-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.161832 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.161827    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd7b42e-bc60-44d5-9359-21ae284f51db-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.518454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.518359    2581 generic.go:356] "Generic (PLEG): container finished" podID="22f61d33-1d44-4f5e-b88c-290ebb242cad" containerID="57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9" exitCode=0
Sep 09 18:53:20.518454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.518419    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"
Sep 09 18:53:20.518454 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.518442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" event={"ID":"22f61d33-1d44-4f5e-b88c-290ebb242cad","Type":"ContainerDied","Data":"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"}
Sep 09 18:53:20.518726 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.518503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28" event={"ID":"22f61d33-1d44-4f5e-b88c-290ebb242cad","Type":"ContainerDied","Data":"56e2dded2691e8814179f3c4083850613042e1b115bf20e272e51408538854ff"}
Sep 09 18:53:20.518726 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.518532    2581 scope.go:117] "RemoveContainer" containerID="57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"
Sep 09 18:53:20.519918 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.519893    2581 generic.go:356] "Generic (PLEG): container finished" podID="cbd7b42e-bc60-44d5-9359-21ae284f51db" containerID="f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b" exitCode=0
Sep 09 18:53:20.520048 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.519952    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"
Sep 09 18:53:20.520048 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.519995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" event={"ID":"cbd7b42e-bc60-44d5-9359-21ae284f51db","Type":"ContainerDied","Data":"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"}
Sep 09 18:53:20.520149 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.520051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj" event={"ID":"cbd7b42e-bc60-44d5-9359-21ae284f51db","Type":"ContainerDied","Data":"9d3884d92cf8f4fee07a44e6a8de852078a8c4243e56d8b427e0153b4c75d3d9"}
Sep 09 18:53:20.527518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.527504    2581 scope.go:117] "RemoveContainer" containerID="57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"
Sep 09 18:53:20.527816 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:53:20.527791    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9\": container with ID starting with 57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9 not found: ID does not exist" containerID="57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"
Sep 09 18:53:20.527920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.527819    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9"} err="failed to get container status \"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9\": rpc error: code = NotFound desc = could not find container \"57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9\": container with ID starting with 57f78d17b155f1aafe071c7855e5c1b8d4eadabef47451948559f9a920b25bc9 not found: ID does not exist"
Sep 09 18:53:20.527920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.527836    2581 scope.go:117] "RemoveContainer" containerID="f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"
Sep 09 18:53:20.535325 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.535312    2581 scope.go:117] "RemoveContainer" containerID="f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"
Sep 09 18:53:20.535581 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:53:20.535564    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b\": container with ID starting with f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b not found: ID does not exist" containerID="f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"
Sep 09 18:53:20.535634 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.535588    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b"} err="failed to get container status \"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b\": rpc error: code = NotFound desc = could not find container \"f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b\": container with ID starting with f13476e1e577585c09f155794659718575c09978e3ffda0796c00d0dead06f9b not found: ID does not exist"
Sep 09 18:53:20.538483 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.538464    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:53:20.545557 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.545538    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ddaae1a6-f6785c9d4-6sjzj"]
Sep 09 18:53:20.556717 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.556679    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:53:20.560819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:20.560800    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ddaae1a6-7f6799b9b9-vlf28"]
Sep 09 18:53:21.192632 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:21.192593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:22.284234 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:22.284198    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f61d33-1d44-4f5e-b88c-290ebb242cad" path="/var/lib/kubelet/pods/22f61d33-1d44-4f5e-b88c-290ebb242cad/volumes"
Sep 09 18:53:22.284614 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:22.284597    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd7b42e-bc60-44d5-9359-21ae284f51db" path="/var/lib/kubelet/pods/cbd7b42e-bc60-44d5-9359-21ae284f51db/volumes"
Sep 09 18:53:24.507878 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:24.507837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:27.076748 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:27.076689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:27.171118 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:27.171085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:27.323222 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:27.323188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:27.536397 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:27.536359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:29.796872 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:29.796842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:33.185459 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:33.185424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:33.410404 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:33.410368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:33.515893 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:33.515814    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:33.610421 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:33.610390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:40.297251 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:40.297217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:40.397257 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:40.397219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:40.558961 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:40.558869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:40.991678 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:40.991647    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:41.434053 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:41.434022    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:45.496544 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.496504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:45.595504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.595463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:45.958894 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.958863    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:53:45.959296 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd7b42e-bc60-44d5-9359-21ae284f51db" containerName="maas-api"
Sep 09 18:53:45.959361 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959298    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd7b42e-bc60-44d5-9359-21ae284f51db" containerName="maas-api"
Sep 09 18:53:45.959361 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959329    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f61d33-1d44-4f5e-b88c-290ebb242cad" containerName="payload-processing"
Sep 09 18:53:45.959361 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959338    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f61d33-1d44-4f5e-b88c-290ebb242cad" containerName="payload-processing"
Sep 09 18:53:45.959459 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959393    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd7b42e-bc60-44d5-9359-21ae284f51db" containerName="maas-api"
Sep 09 18:53:45.959459 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.959407    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f61d33-1d44-4f5e-b88c-290ebb242cad" containerName="payload-processing"
Sep 09 18:53:45.964105 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.964086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:45.966694 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.966669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-0daf23-serving-cert\""
Sep 09 18:53:45.974114 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:45.974090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:53:46.004313 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.004283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:46.086114 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.086077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.086301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.086167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.086301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.086218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.086301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.086260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x462z\" (UniqueName: \"kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.105029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.104999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:46.187022 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.186988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.187221 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.187060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.187221 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.187094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.187221 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.187115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x462z\" (UniqueName: \"kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.187221 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:53:46.187162    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-serving-cert: secret "maas-api-e2e-shared-a-w2-0daf23-serving-cert" not found
Sep 09 18:53:46.187427 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:53:46.187242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls podName:5b8ce878-09d5-42b8-9836-a3354cc65511 nodeName:}" failed. No retries permitted until 2026-09-09 18:53:46.687219839 +0000 UTC m=+1332.918826179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls") pod "maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" (UID: "5b8ce878-09d5-42b8-9836-a3354cc65511") : secret "maas-api-e2e-shared-a-w2-0daf23-serving-cert" not found
Sep 09 18:53:46.187843 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.187819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.189617 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.189582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.195692 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.195672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x462z\" (UniqueName: \"kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.298058 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.297960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:46.494345 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.494317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:46.594081 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.593987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:53:46.598069 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.598044    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:53:46.601592 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.601577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.604060 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.604042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0daf23\""
Sep 09 18:53:46.614930 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.614907    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:53:46.637180 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.637156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:53:46.640419 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.640401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.643032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.643013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0daf23-dockercfg-z8hp5\""
Sep 09 18:53:46.652623 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.652602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:53:46.691819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.691783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjzp\" (UniqueName: \"kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.691819 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.691817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.692015 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.691837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.692015 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.691974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.692084 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.692013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.694482 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.694465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.793287 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.793245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.793472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.793350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjzp\" (UniqueName: \"kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.793472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.793385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.793472 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.793412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.794087 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.794062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.794193 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.794089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.803889 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.803865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh\") pod \"payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.803989 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.803967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjzp\" (UniqueName: \"kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp\") pod \"payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:46.875686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.875648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:46.912164 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.912126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:46.951250 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:46.951216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:47.162341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.162314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:53:47.164252 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:53:47.164230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8ce878_09d5_42b8_9836_a3354cc65511.slice/crio-0d8912e6028f5029842415f86a2267927f1322aa327e533f38ab79ea396a7167 WatchSource:0}: Error finding container 0d8912e6028f5029842415f86a2267927f1322aa327e533f38ab79ea396a7167: Status 404 returned error can't find the container with id 0d8912e6028f5029842415f86a2267927f1322aa327e533f38ab79ea396a7167
Sep 09 18:53:47.224006 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.223983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:53:47.226396 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:53:47.226372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd5d65e_3b09_4b02_92c4_fc89ce1c3b37.slice/crio-3875c5906df28eae04412ec83322f07aba278a2a69dcb97bb16b8c68f1be769d WatchSource:0}: Error finding container 3875c5906df28eae04412ec83322f07aba278a2a69dcb97bb16b8c68f1be769d: Status 404 returned error can't find the container with id 3875c5906df28eae04412ec83322f07aba278a2a69dcb97bb16b8c68f1be769d
Sep 09 18:53:47.247448 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.247428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:53:47.249522 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:53:47.249499    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f29897e_a5cd_4ea7_ae59_43abdcdb8107.slice/crio-fc7c024340bc2b4abf800633b800422438ca3d343a68c6dadb7a96b01ed88247 WatchSource:0}: Error finding container fc7c024340bc2b4abf800633b800422438ca3d343a68c6dadb7a96b01ed88247: Status 404 returned error can't find the container with id fc7c024340bc2b4abf800633b800422438ca3d343a68c6dadb7a96b01ed88247
Sep 09 18:53:47.645415 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.645383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" event={"ID":"3f29897e-a5cd-4ea7-ae59-43abdcdb8107","Type":"ContainerStarted","Data":"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"}
Sep 09 18:53:47.645864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.645431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" event={"ID":"3f29897e-a5cd-4ea7-ae59-43abdcdb8107","Type":"ContainerStarted","Data":"fc7c024340bc2b4abf800633b800422438ca3d343a68c6dadb7a96b01ed88247"}
Sep 09 18:53:47.645864 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.645468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:47.646755 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.646733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" event={"ID":"5b8ce878-09d5-42b8-9836-a3354cc65511","Type":"ContainerStarted","Data":"0d8912e6028f5029842415f86a2267927f1322aa327e533f38ab79ea396a7167"}
Sep 09 18:53:47.648140 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.648115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" event={"ID":"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37","Type":"ContainerStarted","Data":"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"}
Sep 09 18:53:47.648280 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.648150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" event={"ID":"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37","Type":"ContainerStarted","Data":"3875c5906df28eae04412ec83322f07aba278a2a69dcb97bb16b8c68f1be769d"}
Sep 09 18:53:47.648341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.648298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:53:47.670141 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.670093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" podStartSLOduration=1.670077738 podStartE2EDuration="1.670077738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:47.662246924 +0000 UTC m=+1333.893853289" watchObservedRunningTime="2026-09-09 18:53:47.670077738 +0000 UTC m=+1333.901684146"
Sep 09 18:53:47.681126 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:47.681084    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" podStartSLOduration=1.681072873 podStartE2EDuration="1.681072873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:47.67895408 +0000 UTC m=+1333.910560452" watchObservedRunningTime="2026-09-09 18:53:47.681072873 +0000 UTC m=+1333.912679281"
Sep 09 18:53:49.661035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:49.660988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" event={"ID":"5b8ce878-09d5-42b8-9836-a3354cc65511","Type":"ContainerStarted","Data":"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"}
Sep 09 18:53:49.661481 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:49.661124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:49.678237 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:49.678189    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" podStartSLOduration=3.25887262 podStartE2EDuration="4.678176138s" totalImagesPullingTime="1.419303518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:49.676933326 +0000 UTC m=+1335.908539688" watchObservedRunningTime="2026-09-09 18:53:49.678176138 +0000 UTC m=+1335.909782499"
Sep 09 18:53:55.671784 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:55.671691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:53:58.658083 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:58.658056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:53:58.658504 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:53:58.658121    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:54:09.766745 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:09.766685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:11.026368 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:11.026340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:11.213276 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:11.213231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:13.824845 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:13.824812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:13.914858 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:13.914827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:14.944364 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:14.944330    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 18:54:14.948377 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:14.948354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:14.951213 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:14.951189    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c09a03-serving-cert\""
Sep 09 18:54:14.958250 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:14.958106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 18:54:15.016914 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.016880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:15.049920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.049881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.049920 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.049918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nmj\" (UniqueName: \"kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.050152 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.049946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.050152 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.050107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.126348 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.126319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:15.150804 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.150776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.150932 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.150829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.150932 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.150848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nmj\" (UniqueName: \"kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.150932 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.150877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.150932 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:54:15.150911    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-serving-cert: secret "maas-api-e2e-shared-b-w2-c09a03-serving-cert" not found
Sep 09 18:54:15.151104 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:54:15.150980    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls podName:db5efe83-c5aa-4c9a-924f-9254eaeccdd8 nodeName:}" failed. No retries permitted until 2026-09-09 18:54:15.650964781 +0000 UTC m=+1361.882571121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" (UID: "db5efe83-c5aa-4c9a-924f-9254eaeccdd8") : secret "maas-api-e2e-shared-b-w2-c09a03-serving-cert" not found
Sep 09 18:54:15.151603 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.151556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.153435 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.153414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.164761 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.164738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nmj\" (UniqueName: \"kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.430435 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.430394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:15.527522 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.527489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:15.603401 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.603367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:15.604478 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.604459    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 18:54:15.608035 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.608019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.610600 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.610545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c09a03-dockercfg-gwnj8\""
Sep 09 18:54:15.610732 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.610696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c09a03\""
Sep 09 18:54:15.619958 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.619935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 18:54:15.654170 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.654142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.656586 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.656562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.755498 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.755398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.755672 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.755530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwq7z\" (UniqueName: \"kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.856862 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.856826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwq7z\" (UniqueName: \"kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.857048 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.856872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.857576 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.857557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.861763 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.861735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:15.865399 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.865377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwq7z\" (UniqueName: \"kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z\") pod \"payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:15.920249 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:15.920020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:16.136767 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.136732    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 18:54:16.139953 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:54:16.139923    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5efe83_c5aa_4c9a_924f_9254eaeccdd8.slice/crio-01b3ed8e7fc7401b93e9f3a562bdeff9f8c6ca5d59a3aea7ab1407558012e8c3 WatchSource:0}: Error finding container 01b3ed8e7fc7401b93e9f3a562bdeff9f8c6ca5d59a3aea7ab1407558012e8c3: Status 404 returned error can't find the container with id 01b3ed8e7fc7401b93e9f3a562bdeff9f8c6ca5d59a3aea7ab1407558012e8c3
Sep 09 18:54:16.207402 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.207380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 18:54:16.209889 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:54:16.209862    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00eeb81_1f49_4900_84b7_d9e8299b66de.slice/crio-466afa2f8fd472abb0dbdc7a14e3efba5b23509380d5bc7afa170882aa95c945 WatchSource:0}: Error finding container 466afa2f8fd472abb0dbdc7a14e3efba5b23509380d5bc7afa170882aa95c945: Status 404 returned error can't find the container with id 466afa2f8fd472abb0dbdc7a14e3efba5b23509380d5bc7afa170882aa95c945
Sep 09 18:54:16.784287 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.784187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" event={"ID":"db5efe83-c5aa-4c9a-924f-9254eaeccdd8","Type":"ContainerStarted","Data":"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"}
Sep 09 18:54:16.784287 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.784252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" event={"ID":"db5efe83-c5aa-4c9a-924f-9254eaeccdd8","Type":"ContainerStarted","Data":"01b3ed8e7fc7401b93e9f3a562bdeff9f8c6ca5d59a3aea7ab1407558012e8c3"}
Sep 09 18:54:16.784287 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.784283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:16.785907 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.785883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" event={"ID":"c00eeb81-1f49-4900-84b7-d9e8299b66de","Type":"ContainerStarted","Data":"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"}
Sep 09 18:54:16.786046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.785933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:16.786046 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.785956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" event={"ID":"c00eeb81-1f49-4900-84b7-d9e8299b66de","Type":"ContainerStarted","Data":"466afa2f8fd472abb0dbdc7a14e3efba5b23509380d5bc7afa170882aa95c945"}
Sep 09 18:54:16.804754 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.804683    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" podStartSLOduration=2.423650123 podStartE2EDuration="2.804667187s" totalImagesPullingTime="381.017064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:16.802343584 +0000 UTC m=+1363.033949947" watchObservedRunningTime="2026-09-09 18:54:16.804667187 +0000 UTC m=+1363.036273549"
Sep 09 18:54:16.821319 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:16.821256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" podStartSLOduration=1.8212397359999999 podStartE2EDuration="1.821239736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:16.819376821 +0000 UTC m=+1363.050983211" watchObservedRunningTime="2026-09-09 18:54:16.821239736 +0000 UTC m=+1363.052846098"
Sep 09 18:54:22.797104 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:22.797074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 18:54:27.793045 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:27.793007    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 18:54:38.903229 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:38.903196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:48.361718 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:48.361666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:48.659934 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:48.659858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:48.759981 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:48.759951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:52.079410 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:52.079378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:52.654582 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:52.654545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:53.513912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.513879    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:54:53.518406 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.518383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.521619 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.521595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-212790ce-serving-cert\""
Sep 09 18:54:53.531600 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.531575    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:54:53.558624 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.558589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:53.609899 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.609865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.610044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.609911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5zv\" (UniqueName: \"kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.610044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.609988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.610044 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.610018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.704528 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.704496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:53.710725 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.710683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.710853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.710754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5zv\" (UniqueName: \"kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.710853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.710810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.710853 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:54:53.710829    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-serving-cert: secret "maas-api-e2e-mt-212790ce-serving-cert" not found
Sep 09 18:54:53.710853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.710840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.711019 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:54:53.710902    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls podName:e9ea6ec7-1113-4054-811f-61c84232213b nodeName:}" failed. No retries permitted until 2026-09-09 18:54:54.210881481 +0000 UTC m=+1400.442487835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls") pod "maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" (UID: "e9ea6ec7-1113-4054-811f-61c84232213b") : secret "maas-api-e2e-mt-212790ce-serving-cert" not found
Sep 09 18:54:53.711477 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.711458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.713431 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.713408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.727697 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.727674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5zv\" (UniqueName: \"kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:53.872937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:53.872899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:54.216012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.215929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:54.218379 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.218360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") pod \"maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:54.296001 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.295974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:54.410293 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.410259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:54:54.428754 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.428701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:54.706902 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.706877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:54:54.971853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:54.971817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" event={"ID":"e9ea6ec7-1113-4054-811f-61c84232213b","Type":"ContainerStarted","Data":"ae7da46055de71aaef0d744e219323e864fad82ecf29c6c75b654c918f2f4f1f"}
Sep 09 18:54:55.978516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:55.978477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" event={"ID":"e9ea6ec7-1113-4054-811f-61c84232213b","Type":"ContainerStarted","Data":"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"}
Sep 09 18:54:55.978937 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:55.978624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:54:56.001123 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:54:56.001078    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" podStartSLOduration=2.421853329 podStartE2EDuration="3.001061495s" totalImagesPullingTime="579.208166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:55.999567238 +0000 UTC m=+1402.231173599" watchObservedRunningTime="2026-09-09 18:54:56.001061495 +0000 UTC m=+1402.232667857"
Sep 09 18:55:01.989337 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:55:01.989303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:55:17.907414 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:55:17.907377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:55:25.065556 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:55:25.065475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:55:40.316689 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:55:40.316654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:55:51.401831 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:55:51.401799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:06.846074 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:06.846040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:56:06.846638 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:06.846303    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" containerName="maas-api" containerID="cri-o://046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3" gracePeriod=30
Sep 09 18:56:06.886233 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:06.886194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:06.984403 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:06.984360    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.52:8443/health\": dial tcp 10.132.0.52:8443: connect: connection refused"
Sep 09 18:56:07.045205 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.045174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:07.108112 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.107731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:56:07.163609 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.163569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z5zv\" (UniqueName: \"kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv\") pod \"e9ea6ec7-1113-4054-811f-61c84232213b\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") "
Sep 09 18:56:07.163609 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.163607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs\") pod \"e9ea6ec7-1113-4054-811f-61c84232213b\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") "
Sep 09 18:56:07.163906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.163678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca\") pod \"e9ea6ec7-1113-4054-811f-61c84232213b\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") "
Sep 09 18:56:07.163906 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.163754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") pod \"e9ea6ec7-1113-4054-811f-61c84232213b\" (UID: \"e9ea6ec7-1113-4054-811f-61c84232213b\") "
Sep 09 18:56:07.164130 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.164093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9ea6ec7-1113-4054-811f-61c84232213b" (UID: "e9ea6ec7-1113-4054-811f-61c84232213b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:56:07.165984 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.165956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9ea6ec7-1113-4054-811f-61c84232213b" (UID: "e9ea6ec7-1113-4054-811f-61c84232213b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:56:07.166082 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.165990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv" (OuterVolumeSpecName: "kube-api-access-6z5zv") pod "e9ea6ec7-1113-4054-811f-61c84232213b" (UID: "e9ea6ec7-1113-4054-811f-61c84232213b"). InnerVolumeSpecName "kube-api-access-6z5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:07.166314 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.166300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9ea6ec7-1113-4054-811f-61c84232213b" (UID: "e9ea6ec7-1113-4054-811f-61c84232213b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:56:07.210422 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.210384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:07.266115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.266025    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ea6ec7-1113-4054-811f-61c84232213b-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.266115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.266060    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.266115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.266073    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5zv\" (UniqueName: \"kubernetes.io/projected/e9ea6ec7-1113-4054-811f-61c84232213b-kube-api-access-6z5zv\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.266115 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.266085    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9ea6ec7-1113-4054-811f-61c84232213b-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.287290 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.287263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:07.321147 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.321110    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9ea6ec7-1113-4054-811f-61c84232213b" containerID="046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3" exitCode=0
Sep 09 18:56:07.321335 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.321163    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"
Sep 09 18:56:07.321335 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.321167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" event={"ID":"e9ea6ec7-1113-4054-811f-61c84232213b","Type":"ContainerDied","Data":"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"}
Sep 09 18:56:07.321335 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.321278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv" event={"ID":"e9ea6ec7-1113-4054-811f-61c84232213b","Type":"ContainerDied","Data":"ae7da46055de71aaef0d744e219323e864fad82ecf29c6c75b654c918f2f4f1f"}
Sep 09 18:56:07.321335 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.321299    2581 scope.go:117] "RemoveContainer" containerID="046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"
Sep 09 18:56:07.330944 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.330925    2581 scope.go:117] "RemoveContainer" containerID="046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"
Sep 09 18:56:07.331193 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:56:07.331177    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3\": container with ID starting with 046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3 not found: ID does not exist" containerID="046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"
Sep 09 18:56:07.331244 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.331201    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3"} err="failed to get container status \"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3\": rpc error: code = NotFound desc = could not find container \"046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3\": container with ID starting with 046edf7f2bf6c6bcd77c66505ad1946cab7123e26bf3882b410ef5d884231dd3 not found: ID does not exist"
Sep 09 18:56:07.343012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.342978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:56:07.346576 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:07.346551    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-212790ce-6575ffbf44-rvdqv"]
Sep 09 18:56:08.285197 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:08.285164    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" path="/var/lib/kubelet/pods/e9ea6ec7-1113-4054-811f-61c84232213b/volumes"
Sep 09 18:56:10.549885 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:10.549846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:10.713917 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:10.713879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:11.811204 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:11.811162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:15.628659 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:15.628632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:32.780945 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:32.780912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:52.187339 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:52.187308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:52.761308 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:52.761274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:56:58.349692 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:56:58.349607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:01.754869 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:01.754836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:02.957332 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:02.957269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:07.739374 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:07.739347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:07.839845 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:07.839815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:08.398483 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:08.398456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:14.419655 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:14.419614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:14.695301 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:14.695221    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:14.801941 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:14.801910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:18.431863 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:18.431827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:25.167369 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:25.167335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:25.393521 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:25.393488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:31.457488 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:31.457439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:46.303974 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:46.303941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:52.608843 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:52.608805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:52.672814 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:52.672785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:52.949029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:52.948950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:53.243032 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:53.242957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:53.427029 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:53.426996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:56.301909 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:56.301837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:56.400400 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:56.400364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:57.447699 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.447660    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:57:57.448294 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.448272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" containerName="maas-api"
Sep 09 18:57:57.448339 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.448299    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" containerName="maas-api"
Sep 09 18:57:57.448415 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.448403    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ea6ec7-1113-4054-811f-61c84232213b" containerName="maas-api"
Sep 09 18:57:57.451913 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.451895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.454462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.454439    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-79b1ac-serving-cert\""
Sep 09 18:57:57.463463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.463416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:57:57.487024 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.486978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:57.536731 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.536684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.536881 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.536761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp695\" (UniqueName: \"kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.536881 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.536791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.536881 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.536823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.587510 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.587483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:57.638139 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.638113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.638316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.638188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.638316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.638225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp695\" (UniqueName: \"kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.638316 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.638248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.638553 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:57:57.638335    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-serving-cert: secret "maas-api-e2e-api-a-79b1ac-serving-cert" not found
Sep 09 18:57:57.638553 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:57:57.638393    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls podName:97f81089-0379-4b5e-888b-81e6e16efdbe nodeName:}" failed. No retries permitted until 2026-09-09 18:57:58.138376003 +0000 UTC m=+1584.369982343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls") pod "maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" (UID: "97f81089-0379-4b5e-888b-81e6e16efdbe") : secret "maas-api-e2e-api-a-79b1ac-serving-cert" not found
Sep 09 18:57:57.638891 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.638870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.640669 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.640651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.646905 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.646883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp695\" (UniqueName: \"kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:57.986985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:57.986949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:58.085353 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.085315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:57:58.142902 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.142870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:58.145355 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.145323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:58.363753 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.363699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:58.642754 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.642637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:57:58.645129 ip-10-0-135-104 kubenswrapper[2581]: W0909 18:57:58.645089    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f81089_0379_4b5e_888b_81e6e16efdbe.slice/crio-afe88b05ff45cc9d97bfcb1beb6475c7c32646a49e421a5020bffa68a7d86b47 WatchSource:0}: Error finding container afe88b05ff45cc9d97bfcb1beb6475c7c32646a49e421a5020bffa68a7d86b47: Status 404 returned error can't find the container with id afe88b05ff45cc9d97bfcb1beb6475c7c32646a49e421a5020bffa68a7d86b47
Sep 09 18:57:58.646505 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.646489    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:57:58.844778 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:58.844742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" event={"ID":"97f81089-0379-4b5e-888b-81e6e16efdbe","Type":"ContainerStarted","Data":"afe88b05ff45cc9d97bfcb1beb6475c7c32646a49e421a5020bffa68a7d86b47"}
Sep 09 18:57:59.852012 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:59.851971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" event={"ID":"97f81089-0379-4b5e-888b-81e6e16efdbe","Type":"ContainerStarted","Data":"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"}
Sep 09 18:57:59.852443 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:59.852058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:57:59.870697 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:57:59.870606    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" podStartSLOduration=1.9110335109999999 podStartE2EDuration="2.870591058s" totalImagesPullingTime="959.557547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:57:59.869116695 +0000 UTC m=+1586.100723079" watchObservedRunningTime="2026-09-09 18:57:59.870591058 +0000 UTC m=+1586.102197420"
Sep 09 18:58:05.862991 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:05.862962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:58:13.039490 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:13.039460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:17.266138 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:17.266068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:18.215101 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:18.215070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:21.200466 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:21.200435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:21.663880 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:21.663849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:21.975566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:21.975488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:22.368139 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:22.368108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:25.199133 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:25.199059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:25.434735 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:25.434670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:25.528324 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:25.528232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:26.749516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:26.749488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:26.854614 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:26.854573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:27.013969 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:27.013879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:27.566915 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:27.566880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:27.662836 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:27.662806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:31.796820 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:31.796782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:31.856879 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:31.856849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:50.368943 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:50.368911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:58:53.975686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:58:53.975645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:04.156717 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:04.156665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:07.498776 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:07.498744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:08.755737 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:08.755682    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:59:08.756192 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:08.756038    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" podUID="97f81089-0379-4b5e-888b-81e6e16efdbe" containerName="maas-api" containerID="cri-o://0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d" gracePeriod=30
Sep 09 18:59:08.785853 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:08.785815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:09.012436 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.012380    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:59:09.175898 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.175868    2581 generic.go:356] "Generic (PLEG): container finished" podID="97f81089-0379-4b5e-888b-81e6e16efdbe" containerID="0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d" exitCode=0
Sep 09 18:59:09.176070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.175905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"
Sep 09 18:59:09.176070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.175943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" event={"ID":"97f81089-0379-4b5e-888b-81e6e16efdbe","Type":"ContainerDied","Data":"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"}
Sep 09 18:59:09.176070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.175992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62" event={"ID":"97f81089-0379-4b5e-888b-81e6e16efdbe","Type":"ContainerDied","Data":"afe88b05ff45cc9d97bfcb1beb6475c7c32646a49e421a5020bffa68a7d86b47"}
Sep 09 18:59:09.176070 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.176019    2581 scope.go:117] "RemoveContainer" containerID="0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"
Sep 09 18:59:09.186994 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.186933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:09.189968 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.189939    2581 scope.go:117] "RemoveContainer" containerID="0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"
Sep 09 18:59:09.190339 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:59:09.190316    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d\": container with ID starting with 0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d not found: ID does not exist" containerID="0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"
Sep 09 18:59:09.190463 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.190347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d"} err="failed to get container status \"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d\": rpc error: code = NotFound desc = could not find container \"0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d\": container with ID starting with 0192583b5bb44109dcba9294031a0193160ef9a534495a2bad8745a81fcc1c2d not found: ID does not exist"
Sep 09 18:59:09.192691 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.192673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") pod \"97f81089-0379-4b5e-888b-81e6e16efdbe\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") "
Sep 09 18:59:09.192817 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.192734    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca\") pod \"97f81089-0379-4b5e-888b-81e6e16efdbe\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") "
Sep 09 18:59:09.192817 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.192761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs\") pod \"97f81089-0379-4b5e-888b-81e6e16efdbe\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") "
Sep 09 18:59:09.192817 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.192801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp695\" (UniqueName: \"kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695\") pod \"97f81089-0379-4b5e-888b-81e6e16efdbe\" (UID: \"97f81089-0379-4b5e-888b-81e6e16efdbe\") "
Sep 09 18:59:09.193093 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.193070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97f81089-0379-4b5e-888b-81e6e16efdbe" (UID: "97f81089-0379-4b5e-888b-81e6e16efdbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:09.194989 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.194964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97f81089-0379-4b5e-888b-81e6e16efdbe" (UID: "97f81089-0379-4b5e-888b-81e6e16efdbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:59:09.195084 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.195003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97f81089-0379-4b5e-888b-81e6e16efdbe" (UID: "97f81089-0379-4b5e-888b-81e6e16efdbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:59:09.195084 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.195010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695" (OuterVolumeSpecName: "kube-api-access-xp695") pod "97f81089-0379-4b5e-888b-81e6e16efdbe" (UID: "97f81089-0379-4b5e-888b-81e6e16efdbe"). InnerVolumeSpecName "kube-api-access-xp695". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:09.288656 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.288507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:09.293366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.293340    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp695\" (UniqueName: \"kubernetes.io/projected/97f81089-0379-4b5e-888b-81e6e16efdbe-kube-api-access-xp695\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.293366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.293363    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.293366 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.293373    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f81089-0379-4b5e-888b-81e6e16efdbe-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.293560 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.293382    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f81089-0379-4b5e-888b-81e6e16efdbe-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.386936 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.386903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:09.498566 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.498535    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:59:09.502653 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:09.502632    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79b1ac-bcd5978cf-clz62"]
Sep 09 18:59:10.284444 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:10.284404    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f81089-0379-4b5e-888b-81e6e16efdbe" path="/var/lib/kubelet/pods/97f81089-0379-4b5e-888b-81e6e16efdbe/volumes"
Sep 09 18:59:15.268112 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:15.268077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:15.368517 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:15.368483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:15.598318 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:15.598216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:16.344461 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:16.344430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:18.339464 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:18.339403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:18.871912 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:18.871876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:23.336161 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:23.336125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:31.420076 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:31.420044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:31.493848 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:31.493815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:31.568318 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:31.568284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:31.632669 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:31.632635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:31.695590 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:31.695503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:32.004972 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:32.004885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:36.350462 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:36.350411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:38.177840 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:38.177808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:38.251244 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:38.251214    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:39.619138 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:39.619109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:39.719883 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:39.719852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.262881 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.262839    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:59:49.263350 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.263088    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" podUID="5b8ce878-09d5-42b8-9836-a3354cc65511" containerName="maas-api" containerID="cri-o://4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601" gracePeriod=30
Sep 09 18:59:49.285747 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.284135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.329353 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.329324    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:59:49.329644 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.329569    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" podUID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" containerName="payload-processing" containerID="cri-o://3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da" gracePeriod=30
Sep 09 18:59:49.335639 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.335612    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:59:49.335904 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.335879    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" podUID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" containerName="payload-pre-processing" containerID="cri-o://72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c" gracePeriod=30
Sep 09 18:59:49.371821 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.371793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.444130 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.444065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.509961 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.509928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.589205 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.589141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.629043 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.629022    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:59:49.632207 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.632171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:59:49.638833 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.638814    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:59:49.758244 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcjzp\" (UniqueName: \"kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp\") pod \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") "
Sep 09 18:59:49.758244 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") pod \"5b8ce878-09d5-42b8-9836-a3354cc65511\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca\") pod \"5b8ce878-09d5-42b8-9836-a3354cc65511\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs\") pod \"5b8ce878-09d5-42b8-9836-a3354cc65511\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume\") pod \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\" (UID: \"3f29897e-a5cd-4ea7-ae59-43abdcdb8107\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh\") pod \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume\") pod \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\" (UID: \"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37\") "
Sep 09 18:59:49.758518 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x462z\" (UniqueName: \"kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z\") pod \"5b8ce878-09d5-42b8-9836-a3354cc65511\" (UID: \"5b8ce878-09d5-42b8-9836-a3354cc65511\") "
Sep 09 18:59:49.758866 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b8ce878-09d5-42b8-9836-a3354cc65511" (UID: "5b8ce878-09d5-42b8-9836-a3354cc65511"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:49.758866 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f29897e-a5cd-4ea7-ae59-43abdcdb8107" (UID: "3f29897e-a5cd-4ea7-ae59-43abdcdb8107"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:49.758978 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758888    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8ce878-09d5-42b8-9836-a3354cc65511-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.758978 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.758909    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.759248 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.759153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" (UID: "1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:49.760683 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.760654    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp" (OuterVolumeSpecName: "kube-api-access-wcjzp") pod "3f29897e-a5cd-4ea7-ae59-43abdcdb8107" (UID: "3f29897e-a5cd-4ea7-ae59-43abdcdb8107"). InnerVolumeSpecName "kube-api-access-wcjzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:49.760829 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.760681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b8ce878-09d5-42b8-9836-a3354cc65511" (UID: "5b8ce878-09d5-42b8-9836-a3354cc65511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:59:49.760829 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.760700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b8ce878-09d5-42b8-9836-a3354cc65511" (UID: "5b8ce878-09d5-42b8-9836-a3354cc65511"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:59:49.761503 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.761480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh" (OuterVolumeSpecName: "kube-api-access-q5wjh") pod "1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" (UID: "1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37"). InnerVolumeSpecName "kube-api-access-q5wjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:49.761570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.761528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z" (OuterVolumeSpecName: "kube-api-access-x462z") pod "5b8ce878-09d5-42b8-9836-a3354cc65511" (UID: "5b8ce878-09d5-42b8-9836-a3354cc65511"). InnerVolumeSpecName "kube-api-access-x462z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:49.855334 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.855307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:49.859443 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859423    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.859533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859445    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x462z\" (UniqueName: \"kubernetes.io/projected/5b8ce878-09d5-42b8-9836-a3354cc65511-kube-api-access-x462z\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.859533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859456    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcjzp\" (UniqueName: \"kubernetes.io/projected/3f29897e-a5cd-4ea7-ae59-43abdcdb8107-kube-api-access-wcjzp\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.859533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859467    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.859533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859475    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8ce878-09d5-42b8-9836-a3354cc65511-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:49.859533 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:49.859482    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37-kube-api-access-q5wjh\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 18:59:50.375111 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.375074    2581 generic.go:356] "Generic (PLEG): container finished" podID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" containerID="72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c" exitCode=0
Sep 09 18:59:50.375570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.375116    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"
Sep 09 18:59:50.375570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.375162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" event={"ID":"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37","Type":"ContainerDied","Data":"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"}
Sep 09 18:59:50.375570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.375218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk" event={"ID":"1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37","Type":"ContainerDied","Data":"3875c5906df28eae04412ec83322f07aba278a2a69dcb97bb16b8c68f1be769d"}
Sep 09 18:59:50.375570 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.375239    2581 scope.go:117] "RemoveContainer" containerID="72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"
Sep 09 18:59:50.376665 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.376642    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" containerID="3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da" exitCode=0
Sep 09 18:59:50.376773 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.376692    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"
Sep 09 18:59:50.376773 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.376750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" event={"ID":"3f29897e-a5cd-4ea7-ae59-43abdcdb8107","Type":"ContainerDied","Data":"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"}
Sep 09 18:59:50.376926 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.376908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp" event={"ID":"3f29897e-a5cd-4ea7-ae59-43abdcdb8107","Type":"ContainerDied","Data":"fc7c024340bc2b4abf800633b800422438ca3d343a68c6dadb7a96b01ed88247"}
Sep 09 18:59:50.378231 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.378209    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b8ce878-09d5-42b8-9836-a3354cc65511" containerID="4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601" exitCode=0
Sep 09 18:59:50.378341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.378254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"
Sep 09 18:59:50.378341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.378285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" event={"ID":"5b8ce878-09d5-42b8-9836-a3354cc65511","Type":"ContainerDied","Data":"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"}
Sep 09 18:59:50.378341 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.378334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj" event={"ID":"5b8ce878-09d5-42b8-9836-a3354cc65511","Type":"ContainerDied","Data":"0d8912e6028f5029842415f86a2267927f1322aa327e533f38ab79ea396a7167"}
Sep 09 18:59:50.385686 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.385670    2581 scope.go:117] "RemoveContainer" containerID="72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"
Sep 09 18:59:50.386030 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:59:50.386011    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c\": container with ID starting with 72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c not found: ID does not exist" containerID="72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"
Sep 09 18:59:50.386119 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.386041    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c"} err="failed to get container status \"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c\": rpc error: code = NotFound desc = could not find container \"72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c\": container with ID starting with 72533e225dffc07fdd9a798922c3cf675cbcc027016634be6b0c554086e2254c not found: ID does not exist"
Sep 09 18:59:50.386119 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.386066    2581 scope.go:117] "RemoveContainer" containerID="3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"
Sep 09 18:59:50.393690 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.393667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:59:50.394102 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.394079    2581 scope.go:117] "RemoveContainer" containerID="3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"
Sep 09 18:59:50.394382 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:59:50.394363    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da\": container with ID starting with 3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da not found: ID does not exist" containerID="3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"
Sep 09 18:59:50.394445 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.394388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da"} err="failed to get container status \"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da\": rpc error: code = NotFound desc = could not find container \"3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da\": container with ID starting with 3fddf6ec4326b92a6917a0bf5b66525a6da1f6892fcff73cb33461a0cec189da not found: ID does not exist"
Sep 09 18:59:50.394445 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.394405    2581 scope.go:117] "RemoveContainer" containerID="4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"
Sep 09 18:59:50.399516 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.399491    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0daf23-86bd98c8bb-qbfzk"]
Sep 09 18:59:50.402990 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.402974    2581 scope.go:117] "RemoveContainer" containerID="4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"
Sep 09 18:59:50.403265 ip-10-0-135-104 kubenswrapper[2581]: E0909 18:59:50.403249    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601\": container with ID starting with 4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601 not found: ID does not exist" containerID="4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"
Sep 09 18:59:50.403312 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.403273    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601"} err="failed to get container status \"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601\": rpc error: code = NotFound desc = could not find container \"4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601\": container with ID starting with 4f4d0fd231a3fe8c51ea71067244d887893fa1a2c13acf8d2d04dfd9d967b601 not found: ID does not exist"
Sep 09 18:59:50.409436 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.409414    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:59:50.413098 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.413077    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0daf23-fd8fff9b9-g2nlj"]
Sep 09 18:59:50.427847 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.426662    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:59:50.432671 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:50.432640    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0daf23-658587768f-fxrfp"]
Sep 09 18:59:52.284231 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:52.284192    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" path="/var/lib/kubelet/pods/1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37/volumes"
Sep 09 18:59:52.284610 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:52.284565    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" path="/var/lib/kubelet/pods/3f29897e-a5cd-4ea7-ae59-43abdcdb8107/volumes"
Sep 09 18:59:52.284985 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:52.284971    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8ce878-09d5-42b8-9836-a3354cc65511" path="/var/lib/kubelet/pods/5b8ce878-09d5-42b8-9836-a3354cc65511/volumes"
Sep 09 18:59:55.609143 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:55.609069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:55.667953 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:55.667919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:55.873810 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:55.873731    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 18:59:56.397034 ip-10-0-135-104 kubenswrapper[2581]: I0909 18:59:56.397002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:06.686576 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.686542    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 19:00:06.687082 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.686896    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" podUID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" containerName="maas-api" containerID="cri-o://f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74" gracePeriod=30
Sep 09 19:00:06.725389 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.725362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:06.754566 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.754532    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 19:00:06.754870 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.754830    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" podUID="c00eeb81-1f49-4900-84b7-d9e8299b66de" containerName="payload-processing" containerID="cri-o://b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a" gracePeriod=30
Sep 09 19:00:06.798074 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.798039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:06.900642 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:06.900600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:07.000150 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.000121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:07.020180 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.020139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 19:00:07.054731 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.054684    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 19:00:07.109689 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109650    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwq7z\" (UniqueName: \"kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z\") pod \"c00eeb81-1f49-4900-84b7-d9e8299b66de\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") "
Sep 09 19:00:07.109927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") pod \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") "
Sep 09 19:00:07.109927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs\") pod \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") "
Sep 09 19:00:07.109927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4nmj\" (UniqueName: \"kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj\") pod \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") "
Sep 09 19:00:07.109927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109868    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca\") pod \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\" (UID: \"db5efe83-c5aa-4c9a-924f-9254eaeccdd8\") "
Sep 09 19:00:07.109927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.109907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume\") pod \"c00eeb81-1f49-4900-84b7-d9e8299b66de\" (UID: \"c00eeb81-1f49-4900-84b7-d9e8299b66de\") "
Sep 09 19:00:07.110342 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.110305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db5efe83-c5aa-4c9a-924f-9254eaeccdd8" (UID: "db5efe83-c5aa-4c9a-924f-9254eaeccdd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:00:07.110459 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.110316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c00eeb81-1f49-4900-84b7-d9e8299b66de" (UID: "c00eeb81-1f49-4900-84b7-d9e8299b66de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:00:07.112096 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.112069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj" (OuterVolumeSpecName: "kube-api-access-r4nmj") pod "db5efe83-c5aa-4c9a-924f-9254eaeccdd8" (UID: "db5efe83-c5aa-4c9a-924f-9254eaeccdd8"). InnerVolumeSpecName "kube-api-access-r4nmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:00:07.112181 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.112093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "db5efe83-c5aa-4c9a-924f-9254eaeccdd8" (UID: "db5efe83-c5aa-4c9a-924f-9254eaeccdd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:00:07.112469 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.112453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z" (OuterVolumeSpecName: "kube-api-access-qwq7z") pod "c00eeb81-1f49-4900-84b7-d9e8299b66de" (UID: "c00eeb81-1f49-4900-84b7-d9e8299b66de"). InnerVolumeSpecName "kube-api-access-qwq7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:00:07.112520 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.112453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db5efe83-c5aa-4c9a-924f-9254eaeccdd8" (UID: "db5efe83-c5aa-4c9a-924f-9254eaeccdd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:00:07.210564 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210528    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00eeb81-1f49-4900-84b7-d9e8299b66de-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.210564 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210556    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwq7z\" (UniqueName: \"kubernetes.io/projected/c00eeb81-1f49-4900-84b7-d9e8299b66de-kube-api-access-qwq7z\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.210564 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210568    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.210827 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210578    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.210827 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4nmj\" (UniqueName: \"kubernetes.io/projected/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-kube-api-access-r4nmj\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.210827 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.210595    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5efe83-c5aa-4c9a-924f-9254eaeccdd8-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.395249 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.395210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:07.456808 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.456771    2581 generic.go:356] "Generic (PLEG): container finished" podID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" containerID="f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74" exitCode=0
Sep 09 19:00:07.456967 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.456818    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"
Sep 09 19:00:07.456967 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.456856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" event={"ID":"db5efe83-c5aa-4c9a-924f-9254eaeccdd8","Type":"ContainerDied","Data":"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"}
Sep 09 19:00:07.456967 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.456929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6" event={"ID":"db5efe83-c5aa-4c9a-924f-9254eaeccdd8","Type":"ContainerDied","Data":"01b3ed8e7fc7401b93e9f3a562bdeff9f8c6ca5d59a3aea7ab1407558012e8c3"}
Sep 09 19:00:07.456967 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.456965    2581 scope.go:117] "RemoveContainer" containerID="f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"
Sep 09 19:00:07.458198 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.458169    2581 generic.go:356] "Generic (PLEG): container finished" podID="c00eeb81-1f49-4900-84b7-d9e8299b66de" containerID="b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a" exitCode=0
Sep 09 19:00:07.458340 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.458196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" event={"ID":"c00eeb81-1f49-4900-84b7-d9e8299b66de","Type":"ContainerDied","Data":"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"}
Sep 09 19:00:07.458340 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.458209    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"
Sep 09 19:00:07.458340 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.458242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc" event={"ID":"c00eeb81-1f49-4900-84b7-d9e8299b66de","Type":"ContainerDied","Data":"466afa2f8fd472abb0dbdc7a14e3efba5b23509380d5bc7afa170882aa95c945"}
Sep 09 19:00:07.466318 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.466207    2581 scope.go:117] "RemoveContainer" containerID="f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"
Sep 09 19:00:07.466626 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:00:07.466607    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74\": container with ID starting with f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74 not found: ID does not exist" containerID="f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"
Sep 09 19:00:07.466745 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.466637    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74"} err="failed to get container status \"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74\": rpc error: code = NotFound desc = could not find container \"f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74\": container with ID starting with f7d559ca92f056574982f7837eed07c26884d35bce9f408a346ac177316acd74 not found: ID does not exist"
Sep 09 19:00:07.466745 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.466659    2581 scope.go:117] "RemoveContainer" containerID="b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"
Sep 09 19:00:07.474537 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.474521    2581 scope.go:117] "RemoveContainer" containerID="b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"
Sep 09 19:00:07.474846 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:00:07.474826    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a\": container with ID starting with b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a not found: ID does not exist" containerID="b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"
Sep 09 19:00:07.474891 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.474854    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a"} err="failed to get container status \"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a\": rpc error: code = NotFound desc = could not find container \"b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a\": container with ID starting with b635b1f344936a0702f320b02bdb09dae2acbc779e54c218fe2d48ec0a9f044a not found: ID does not exist"
Sep 09 19:00:07.481182 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.481161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 19:00:07.487036 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.487012    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c09a03-c8d4b9fc4-g89g6"]
Sep 09 19:00:07.500353 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.500330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 19:00:07.504420 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:07.504401    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c09a03-694ddffccf-z5bqc"]
Sep 09 19:00:08.284247 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:08.284213    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00eeb81-1f49-4900-84b7-d9e8299b66de" path="/var/lib/kubelet/pods/c00eeb81-1f49-4900-84b7-d9e8299b66de/volumes"
Sep 09 19:00:08.284615 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:08.284576    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" path="/var/lib/kubelet/pods/db5efe83-c5aa-4c9a-924f-9254eaeccdd8/volumes"
Sep 09 19:00:08.653927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:08.653885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:09.668539 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:09.668502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:10.478435 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:10.478403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:10.579156 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:10.579125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:10.680004 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:10.679970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:13.292114 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:13.292077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:13.394192 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:13.394161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:13.732657 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:13.732607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:13.831155 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:13.831125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:13.927108 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:13.927078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:14.558664 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.558627    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.558987    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" containerName="maas-api"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.558998    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" containerName="maas-api"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559008    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00eeb81-1f49-4900-84b7-d9e8299b66de" containerName="payload-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559014    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00eeb81-1f49-4900-84b7-d9e8299b66de" containerName="payload-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559023    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" containerName="payload-pre-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559030    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" containerName="payload-pre-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559038    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f81089-0379-4b5e-888b-81e6e16efdbe" containerName="maas-api"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559044    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f81089-0379-4b5e-888b-81e6e16efdbe" containerName="maas-api"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559050    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" containerName="payload-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" containerName="payload-processing"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559064    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8ce878-09d5-42b8-9836-a3354cc65511" containerName="maas-api"
Sep 09 19:00:14.559076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8ce878-09d5-42b8-9836-a3354cc65511" containerName="maas-api"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559119    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00eeb81-1f49-4900-84b7-d9e8299b66de" containerName="payload-processing"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559127    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd5d65e-3b09-4b02-92c4-fc89ce1c3b37" containerName="payload-pre-processing"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559134    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5efe83-c5aa-4c9a-924f-9254eaeccdd8" containerName="maas-api"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559141    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8ce878-09d5-42b8-9836-a3354cc65511" containerName="maas-api"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559147    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f81089-0379-4b5e-888b-81e6e16efdbe" containerName="maas-api"
Sep 09 19:00:14.559492 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.559152    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f29897e-a5cd-4ea7-ae59-43abdcdb8107" containerName="payload-processing"
Sep 09 19:00:14.563879 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.563858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.567215 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.567193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a63380-serving-cert\""
Sep 09 19:00:14.576063 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.576038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:00:14.591340 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.591317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:14.674009 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.673972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb65k\" (UniqueName: \"kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.674009 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.674010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.674222 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.674114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.674222 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.674157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.774633 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.774593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb65k\" (UniqueName: \"kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.774633 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.774637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.774939 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.774700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.774939 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.774773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.775355 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.775333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.777262 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.777241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.777476 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.777454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.782988 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.782969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb65k\" (UniqueName: \"kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k\") pod \"maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:14.852913 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.852887    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:14.875003 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:14.874973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:15.149021 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.148998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:00:15.151272 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:00:15.151239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624eb4a0_fb68_474e_8dfe_d5b186c0c442.slice/crio-640309d7588974d4295e00856b10a3bd637abf536fe6a42c11b1dc8cf6284323 WatchSource:0}: Error finding container 640309d7588974d4295e00856b10a3bd637abf536fe6a42c11b1dc8cf6284323: Status 404 returned error can't find the container with id 640309d7588974d4295e00856b10a3bd637abf536fe6a42c11b1dc8cf6284323
Sep 09 19:00:15.192399 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.192356    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:00:15.197043 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.197018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.199804 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.199776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a63380\""
Sep 09 19:00:15.199923 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.199827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a63380-dockercfg-ghxqn\""
Sep 09 19:00:15.205253 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.205234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:00:15.224460 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.224429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:00:15.230694 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.230667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.240398 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.240363    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:00:15.280015 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.279986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.280177 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.280022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.280177 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.280061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7g4c\" (UniqueName: \"kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.280177 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.280077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5wr\" (UniqueName: \"kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.381482 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.381447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7g4c\" (UniqueName: \"kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.381482 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.381482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5wr\" (UniqueName: \"kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.381753 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.381544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.381753 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.381563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.382219 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.382188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.382468 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.382442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.389938 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.389914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5wr\" (UniqueName: \"kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr\") pod \"payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.390059 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.390039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7g4c\" (UniqueName: \"kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c\") pod \"payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.456996 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.456911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:15.505068 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.505034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" event={"ID":"624eb4a0-fb68-474e-8dfe-d5b186c0c442","Type":"ContainerStarted","Data":"640309d7588974d4295e00856b10a3bd637abf536fe6a42c11b1dc8cf6284323"}
Sep 09 19:00:15.509280 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.509255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:15.541426 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.541386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:15.557093 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.557063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:15.786417 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.786384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:00:15.787492 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:00:15.787468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311d8e02_ca7e_427c_9349_0a77be9961a1.slice/crio-b1fad1ce1f97430f6d198b74e9752c9721e22898359bc119a73b5e99d1602611 WatchSource:0}: Error finding container b1fad1ce1f97430f6d198b74e9752c9721e22898359bc119a73b5e99d1602611: Status 404 returned error can't find the container with id b1fad1ce1f97430f6d198b74e9752c9721e22898359bc119a73b5e99d1602611
Sep 09 19:00:15.830873 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:15.830846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:00:15.833027 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:00:15.832997    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0efd79d_28ff_460d_b6c8_d0a2e3ac7918.slice/crio-febd7437ab4bed8401df4d5e96d1a6382175c350e016c16e4706db7261c3a4a8 WatchSource:0}: Error finding container febd7437ab4bed8401df4d5e96d1a6382175c350e016c16e4706db7261c3a4a8: Status 404 returned error can't find the container with id febd7437ab4bed8401df4d5e96d1a6382175c350e016c16e4706db7261c3a4a8
Sep 09 19:00:16.511817 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.511784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" event={"ID":"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918","Type":"ContainerStarted","Data":"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"}
Sep 09 19:00:16.512008 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.511951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" event={"ID":"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918","Type":"ContainerStarted","Data":"febd7437ab4bed8401df4d5e96d1a6382175c350e016c16e4706db7261c3a4a8"}
Sep 09 19:00:16.512105 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.512036    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:16.513661 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.513635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" event={"ID":"624eb4a0-fb68-474e-8dfe-d5b186c0c442","Type":"ContainerStarted","Data":"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"}
Sep 09 19:00:16.513845 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.513746    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:16.515560 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.515532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" event={"ID":"311d8e02-ca7e-427c-9349-0a77be9961a1","Type":"ContainerStarted","Data":"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"}
Sep 09 19:00:16.515689 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.515582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" event={"ID":"311d8e02-ca7e-427c-9349-0a77be9961a1","Type":"ContainerStarted","Data":"b1fad1ce1f97430f6d198b74e9752c9721e22898359bc119a73b5e99d1602611"}
Sep 09 19:00:16.515689 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.515602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:16.528769 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.528689    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" podStartSLOduration=1.528674905 podStartE2EDuration="1.528674905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:16.526881486 +0000 UTC m=+1722.758487848" watchObservedRunningTime="2026-09-09 19:00:16.528674905 +0000 UTC m=+1722.760281274"
Sep 09 19:00:16.544328 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.544276    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" podStartSLOduration=1.962049014 podStartE2EDuration="2.544262976s" totalImagesPullingTime="582.213962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:16.543193811 +0000 UTC m=+1722.774800196" watchObservedRunningTime="2026-09-09 19:00:16.544262976 +0000 UTC m=+1722.775869340"
Sep 09 19:00:16.560215 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:16.560164    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" podStartSLOduration=1.560144304 podStartE2EDuration="1.560144304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:16.558699589 +0000 UTC m=+1722.790305951" watchObservedRunningTime="2026-09-09 19:00:16.560144304 +0000 UTC m=+1722.791750672"
Sep 09 19:00:22.526365 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:22.526335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:00:27.522742 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:27.522691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:00:27.523178 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:27.522794    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:00:38.200494 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:38.200437    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:38.881119 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:38.881090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:39.185854 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:39.185763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:39.533810 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:39.533725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:41.258255 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:41.258220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:42.442056 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:42.442023    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:42.528905 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:42.528877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:43.708070 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:43.708037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:43.807991 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:43.807957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:44.150566 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:44.150530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:44.278447 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:44.278413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:44.460175 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:44.460097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:51.356089 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:51.356054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:00:56.998236 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:00:56.998202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:07.405233 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:07.405199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:19.172468 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:19.172437    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:48.316355 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:48.316329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7_c0efd79d-28ff-460d-b6c8-d0a2e3ac7918/payload-processing/0.log"
Sep 09 19:01:48.836864 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:48.836812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:49.204308 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:49.204273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:49.411030 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:49.411001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:49.486922 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:49.486840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:52.064722 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:52.064684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:52.165194 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:52.165159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.148029 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.148001    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:01:53.150917 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.150899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.153922 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.153898    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5ae86e-serving-cert\""
Sep 09 19:01:53.164029 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.163998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:01:53.252193 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.252157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc5tx\" (UniqueName: \"kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.252193 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.252196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.252423 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.252243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.252423 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.252286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.353508 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.353479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.353767 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.353527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc5tx\" (UniqueName: \"kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.353767 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.353560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.353767 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.353661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.353767 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:01:53.353735    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-serving-cert: secret "maas-api-e2e-ipp-cleanup-5ae86e-serving-cert" not found
Sep 09 19:01:53.354022 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:01:53.353830    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls podName:05886091-d1ee-435b-8390-c78bb91ef18b nodeName:}" failed. No retries permitted until 2026-09-09 19:01:53.853805154 +0000 UTC m=+1820.085411514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" (UID: "05886091-d1ee-435b-8390-c78bb91ef18b") : secret "maas-api-e2e-ipp-cleanup-5ae86e-serving-cert" not found
Sep 09 19:01:53.354679 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.354654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.356204 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.356179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.364904 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.364885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc5tx\" (UniqueName: \"kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.438262 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.438174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.540538 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.540505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.648974 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.648941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.751482 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.751403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.843079 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.843044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:01:53.858337 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.858306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.860942 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.860919    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:01:53.861333 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.861314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:53.864270 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.864251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:53.867267 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.867243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5ae86e-dockercfg-zhjxm\""
Sep 09 19:01:53.867361 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.867264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5ae86e\""
Sep 09 19:01:53.885479 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.885455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:01:53.958772 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.958737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:53.958772 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:53.958772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvsj5\" (UniqueName: \"kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.059735 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.059629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.059735 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.059668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvsj5\" (UniqueName: \"kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.060378 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.060343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.061546 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.061525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:54.068514 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.068493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvsj5\" (UniqueName: \"kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5\") pod \"payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.178089 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.178054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:54.337560 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.337535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:01:54.339726 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:01:54.339682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05886091_d1ee_435b_8390_c78bb91ef18b.slice/crio-3e4a653e772a62c0cb8bb2d93398f6df857891071f8c9b6b2583a60698485289 WatchSource:0}: Error finding container 3e4a653e772a62c0cb8bb2d93398f6df857891071f8c9b6b2583a60698485289: Status 404 returned error can't find the container with id 3e4a653e772a62c0cb8bb2d93398f6df857891071f8c9b6b2583a60698485289
Sep 09 19:01:54.460494 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.460471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:01:54.462606 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:01:54.462583    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a2cd66_10d9_404e_9e14_bd0d905e67b3.slice/crio-68a63b235f5b6deeeb11f967e5ea93223a6fc0ffaec4085dd9c751fb7333ba8f WatchSource:0}: Error finding container 68a63b235f5b6deeeb11f967e5ea93223a6fc0ffaec4085dd9c751fb7333ba8f: Status 404 returned error can't find the container with id 68a63b235f5b6deeeb11f967e5ea93223a6fc0ffaec4085dd9c751fb7333ba8f
Sep 09 19:01:54.986330 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.986284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" event={"ID":"05886091-d1ee-435b-8390-c78bb91ef18b","Type":"ContainerStarted","Data":"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"}
Sep 09 19:01:54.986517 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.986352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" event={"ID":"05886091-d1ee-435b-8390-c78bb91ef18b","Type":"ContainerStarted","Data":"3e4a653e772a62c0cb8bb2d93398f6df857891071f8c9b6b2583a60698485289"}
Sep 09 19:01:54.987133 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.987108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:01:54.989169 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.989114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" event={"ID":"65a2cd66-10d9-404e-9e14-bd0d905e67b3","Type":"ContainerStarted","Data":"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"}
Sep 09 19:01:54.989310 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.989174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" event={"ID":"65a2cd66-10d9-404e-9e14-bd0d905e67b3","Type":"ContainerStarted","Data":"68a63b235f5b6deeeb11f967e5ea93223a6fc0ffaec4085dd9c751fb7333ba8f"}
Sep 09 19:01:54.989310 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:54.989290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:01:55.006556 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:01:55.006510    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" podStartSLOduration=1.488469643 podStartE2EDuration="2.006497365s" totalImagesPullingTime="518.027722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:01:55.004775718 +0000 UTC m=+1821.236382081" watchObservedRunningTime="2026-09-09 19:01:55.006497365 +0000 UTC m=+1821.238103781"
Sep 09 19:02:02.004576 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:02.004546    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:02:02.023201 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:02.023143    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" podStartSLOduration=9.023124074 podStartE2EDuration="9.023124074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:01:55.021843701 +0000 UTC m=+1821.253450060" watchObservedRunningTime="2026-09-09 19:02:02.023124074 +0000 UTC m=+1828.254730438"
Sep 09 19:02:05.996755 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:05.996698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:02:13.652909 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:13.652874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:18.386049 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:18.386013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:19.576990 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:19.576956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:19.636954 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:19.636925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:23.692190 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.692156    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:02:23.692788 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.692411    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" podUID="05886091-d1ee-435b-8390-c78bb91ef18b" containerName="maas-api" containerID="cri-o://7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d" gracePeriod=30
Sep 09 19:02:23.732076 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.732048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:23.749785 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.749752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:02:23.750046 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.750019    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" podUID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" containerName="payload-processing" containerID="cri-o://73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e" gracePeriod=30
Sep 09 19:02:23.831237 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.831209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:23.930797 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.930766    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:23.998353 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:23.998323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:24.003529 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.003500    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:02:24.026134 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.025924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:02:24.038537 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs\") pod \"05886091-d1ee-435b-8390-c78bb91ef18b\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") "
Sep 09 19:02:24.038746 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume\") pod \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") "
Sep 09 19:02:24.038746 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") pod \"05886091-d1ee-435b-8390-c78bb91ef18b\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") "
Sep 09 19:02:24.038746 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca\") pod \"05886091-d1ee-435b-8390-c78bb91ef18b\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") "
Sep 09 19:02:24.038746 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvsj5\" (UniqueName: \"kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5\") pod \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\" (UID: \"65a2cd66-10d9-404e-9e14-bd0d905e67b3\") "
Sep 09 19:02:24.038746 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.038701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc5tx\" (UniqueName: \"kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx\") pod \"05886091-d1ee-435b-8390-c78bb91ef18b\" (UID: \"05886091-d1ee-435b-8390-c78bb91ef18b\") "
Sep 09 19:02:24.039088 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.039035    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05886091-d1ee-435b-8390-c78bb91ef18b" (UID: "05886091-d1ee-435b-8390-c78bb91ef18b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:24.039667 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.039627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65a2cd66-10d9-404e-9e14-bd0d905e67b3" (UID: "65a2cd66-10d9-404e-9e14-bd0d905e67b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:24.041289 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.041260    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05886091-d1ee-435b-8390-c78bb91ef18b" (UID: "05886091-d1ee-435b-8390-c78bb91ef18b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:02:24.041389 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.041313    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx" (OuterVolumeSpecName: "kube-api-access-cc5tx") pod "05886091-d1ee-435b-8390-c78bb91ef18b" (UID: "05886091-d1ee-435b-8390-c78bb91ef18b"). InnerVolumeSpecName "kube-api-access-cc5tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:24.041562 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.041536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05886091-d1ee-435b-8390-c78bb91ef18b" (UID: "05886091-d1ee-435b-8390-c78bb91ef18b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:02:24.041661 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.041539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5" (OuterVolumeSpecName: "kube-api-access-lvsj5") pod "65a2cd66-10d9-404e-9e14-bd0d905e67b3" (UID: "65a2cd66-10d9-404e-9e14-bd0d905e67b3"). InnerVolumeSpecName "kube-api-access-lvsj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:24.126951 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.126916    2581 generic.go:356] "Generic (PLEG): container finished" podID="05886091-d1ee-435b-8390-c78bb91ef18b" containerID="7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d" exitCode=0
Sep 09 19:02:24.127136 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.126965    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"
Sep 09 19:02:24.127136 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.127000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" event={"ID":"05886091-d1ee-435b-8390-c78bb91ef18b","Type":"ContainerDied","Data":"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"}
Sep 09 19:02:24.127136 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.127053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc" event={"ID":"05886091-d1ee-435b-8390-c78bb91ef18b","Type":"ContainerDied","Data":"3e4a653e772a62c0cb8bb2d93398f6df857891071f8c9b6b2583a60698485289"}
Sep 09 19:02:24.127136 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.127075    2581 scope.go:117] "RemoveContainer" containerID="7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"
Sep 09 19:02:24.128364 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.128335    2581 generic.go:356] "Generic (PLEG): container finished" podID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" containerID="73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e" exitCode=0
Sep 09 19:02:24.128472 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.128364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" event={"ID":"65a2cd66-10d9-404e-9e14-bd0d905e67b3","Type":"ContainerDied","Data":"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"}
Sep 09 19:02:24.128472 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.128392    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"
Sep 09 19:02:24.128472 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.128406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf" event={"ID":"65a2cd66-10d9-404e-9e14-bd0d905e67b3","Type":"ContainerDied","Data":"68a63b235f5b6deeeb11f967e5ea93223a6fc0ffaec4085dd9c751fb7333ba8f"}
Sep 09 19:02:24.138750 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.138508    2581 scope.go:117] "RemoveContainer" containerID="7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"
Sep 09 19:02:24.139096 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:02:24.138878    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d\": container with ID starting with 7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d not found: ID does not exist" containerID="7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"
Sep 09 19:02:24.139096 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.138913    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d"} err="failed to get container status \"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d\": rpc error: code = NotFound desc = could not find container \"7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d\": container with ID starting with 7f927802b7385f021f7d2e6e7ba3da81e5dbbb54b8e00db83db2a0395231a45d not found: ID does not exist"
Sep 09 19:02:24.139096 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.138935    2581 scope.go:117] "RemoveContainer" containerID="73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"
Sep 09 19:02:24.139601 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139582    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc5tx\" (UniqueName: \"kubernetes.io/projected/05886091-d1ee-435b-8390-c78bb91ef18b-kube-api-access-cc5tx\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.139668 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139605    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.139668 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139622    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a2cd66-10d9-404e-9e14-bd0d905e67b3-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.139668 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139634    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05886091-d1ee-435b-8390-c78bb91ef18b-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.139668 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139647    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05886091-d1ee-435b-8390-c78bb91ef18b-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.139668 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.139659    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvsj5\" (UniqueName: \"kubernetes.io/projected/65a2cd66-10d9-404e-9e14-bd0d905e67b3-kube-api-access-lvsj5\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.146857 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.146839    2581 scope.go:117] "RemoveContainer" containerID="73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"
Sep 09 19:02:24.147110 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:02:24.147090    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e\": container with ID starting with 73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e not found: ID does not exist" containerID="73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"
Sep 09 19:02:24.147167 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.147115    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e"} err="failed to get container status \"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e\": rpc error: code = NotFound desc = could not find container \"73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e\": container with ID starting with 73504cc393a6de53494943ff3b6d5cfbaeec95d7a235ac4d7723f1a74a95cb6e not found: ID does not exist"
Sep 09 19:02:24.160455 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.160434    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:02:24.165413 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.165388    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5ae86e-5d5494bf4b-2hgpc"]
Sep 09 19:02:24.189074 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.189045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:02:24.201485 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.201460    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ae86e-597865fbff-wp5cf"]
Sep 09 19:02:24.284766 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.284666    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05886091-d1ee-435b-8390-c78bb91ef18b" path="/var/lib/kubelet/pods/05886091-d1ee-435b-8390-c78bb91ef18b/volumes"
Sep 09 19:02:24.285071 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.285056    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" path="/var/lib/kubelet/pods/65a2cd66-10d9-404e-9e14-bd0d905e67b3/volumes"
Sep 09 19:02:24.310727 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:24.310691    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:30.198217 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:30.198186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:30.409962 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:30.409929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:30.491526 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:30.491429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:34.271213 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:34.271174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.514768 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.514732    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:02:49.515268 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.514980    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" podUID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" containerName="maas-api" containerID="cri-o://89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8" gracePeriod=30
Sep 09 19:02:49.541550 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.541522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.564196 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.564088    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:02:49.564376 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.564304    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" podUID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" containerName="payload-processing" containerID="cri-o://152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8" gracePeriod=30
Sep 09 19:02:49.570447 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.570082    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:02:49.570447 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.570287    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" podUID="311d8e02-ca7e-427c-9349-0a77be9961a1" containerName="payload-pre-processing" containerID="cri-o://deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e" gracePeriod=30
Sep 09 19:02:49.606225 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.606183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.683582 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.682998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.847074 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.847047    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:02:49.849899 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.849874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.865418 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.865396    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:02:49.877072 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.877049    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:02:49.922858 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.922831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:49.977438 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7g4c\" (UniqueName: \"kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c\") pod \"311d8e02-ca7e-427c-9349-0a77be9961a1\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") "
Sep 09 19:02:49.977600 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume\") pod \"311d8e02-ca7e-427c-9349-0a77be9961a1\" (UID: \"311d8e02-ca7e-427c-9349-0a77be9961a1\") "
Sep 09 19:02:49.977600 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs5wr\" (UniqueName: \"kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr\") pod \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") "
Sep 09 19:02:49.977600 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb65k\" (UniqueName: \"kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k\") pod \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") "
Sep 09 19:02:49.977600 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977559    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls\") pod \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") "
Sep 09 19:02:49.977600 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca\") pod \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") "
Sep 09 19:02:49.977919 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs\") pod \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\" (UID: \"624eb4a0-fb68-474e-8dfe-d5b186c0c442\") "
Sep 09 19:02:49.977919 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume\") pod \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\" (UID: \"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918\") "
Sep 09 19:02:49.977919 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.977878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "311d8e02-ca7e-427c-9349-0a77be9961a1" (UID: "311d8e02-ca7e-427c-9349-0a77be9961a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:49.978257 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.978196    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "624eb4a0-fb68-474e-8dfe-d5b186c0c442" (UID: "624eb4a0-fb68-474e-8dfe-d5b186c0c442"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:49.978257 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.978199    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311d8e02-ca7e-427c-9349-0a77be9961a1-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:49.978257 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.978224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" (UID: "c0efd79d-28ff-460d-b6c8-d0a2e3ac7918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:49.979891 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.979854    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k" (OuterVolumeSpecName: "kube-api-access-zb65k") pod "624eb4a0-fb68-474e-8dfe-d5b186c0c442" (UID: "624eb4a0-fb68-474e-8dfe-d5b186c0c442"). InnerVolumeSpecName "kube-api-access-zb65k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:49.979982 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.979936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c" (OuterVolumeSpecName: "kube-api-access-g7g4c") pod "311d8e02-ca7e-427c-9349-0a77be9961a1" (UID: "311d8e02-ca7e-427c-9349-0a77be9961a1"). InnerVolumeSpecName "kube-api-access-g7g4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:49.980053 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.980034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "624eb4a0-fb68-474e-8dfe-d5b186c0c442" (UID: "624eb4a0-fb68-474e-8dfe-d5b186c0c442"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:02:49.980343 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.980323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "624eb4a0-fb68-474e-8dfe-d5b186c0c442" (UID: "624eb4a0-fb68-474e-8dfe-d5b186c0c442"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:02:49.980343 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:49.980325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr" (OuterVolumeSpecName: "kube-api-access-bs5wr") pod "c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" (UID: "c0efd79d-28ff-460d-b6c8-d0a2e3ac7918"). InnerVolumeSpecName "kube-api-access-bs5wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.078962    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.078995    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-plugins-config-volume\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.079010    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7g4c\" (UniqueName: \"kubernetes.io/projected/311d8e02-ca7e-427c-9349-0a77be9961a1-kube-api-access-g7g4c\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.079024    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5wr\" (UniqueName: \"kubernetes.io/projected/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918-kube-api-access-bs5wr\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.079036    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb65k\" (UniqueName: \"kubernetes.io/projected/624eb4a0-fb68-474e-8dfe-d5b186c0c442-kube-api-access-zb65k\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.079047    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624eb4a0-fb68-474e-8dfe-d5b186c0c442-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.079353 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.079059    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624eb4a0-fb68-474e-8dfe-d5b186c0c442-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:02:50.244452 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.244421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:50.249182 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.249151    2581 generic.go:356] "Generic (PLEG): container finished" podID="311d8e02-ca7e-427c-9349-0a77be9961a1" containerID="deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e" exitCode=0
Sep 09 19:02:50.249330 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.249198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"
Sep 09 19:02:50.249330 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.249197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" event={"ID":"311d8e02-ca7e-427c-9349-0a77be9961a1","Type":"ContainerDied","Data":"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"}
Sep 09 19:02:50.249330 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.249291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp" event={"ID":"311d8e02-ca7e-427c-9349-0a77be9961a1","Type":"ContainerDied","Data":"b1fad1ce1f97430f6d198b74e9752c9721e22898359bc119a73b5e99d1602611"}
Sep 09 19:02:50.249330 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.249308    2581 scope.go:117] "RemoveContainer" containerID="deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"
Sep 09 19:02:50.250855 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.250837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"
Sep 09 19:02:50.250959 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.250873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" event={"ID":"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918","Type":"ContainerDied","Data":"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"}
Sep 09 19:02:50.250959 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.250813    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" containerID="152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8" exitCode=0
Sep 09 19:02:50.251060 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.250992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7" event={"ID":"c0efd79d-28ff-460d-b6c8-d0a2e3ac7918","Type":"ContainerDied","Data":"febd7437ab4bed8401df4d5e96d1a6382175c350e016c16e4706db7261c3a4a8"}
Sep 09 19:02:50.252361 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.252337    2581 generic.go:356] "Generic (PLEG): container finished" podID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" containerID="89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8" exitCode=0
Sep 09 19:02:50.252490 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.252356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" event={"ID":"624eb4a0-fb68-474e-8dfe-d5b186c0c442","Type":"ContainerDied","Data":"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"}
Sep 09 19:02:50.252490 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.252384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"
Sep 09 19:02:50.252490 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.252413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg" event={"ID":"624eb4a0-fb68-474e-8dfe-d5b186c0c442","Type":"ContainerDied","Data":"640309d7588974d4295e00856b10a3bd637abf536fe6a42c11b1dc8cf6284323"}
Sep 09 19:02:50.259613 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.259597    2581 scope.go:117] "RemoveContainer" containerID="deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"
Sep 09 19:02:50.259936 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:02:50.259915    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e\": container with ID starting with deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e not found: ID does not exist" containerID="deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"
Sep 09 19:02:50.260025 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.259942    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e"} err="failed to get container status \"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e\": rpc error: code = NotFound desc = could not find container \"deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e\": container with ID starting with deae614005f17a1f12220d94d32824bf2d48a039f0d8fd3923b4407bfd88783e not found: ID does not exist"
Sep 09 19:02:50.260025 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.259958    2581 scope.go:117] "RemoveContainer" containerID="152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"
Sep 09 19:02:50.268092 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.268077    2581 scope.go:117] "RemoveContainer" containerID="152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"
Sep 09 19:02:50.268338 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:02:50.268319    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8\": container with ID starting with 152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8 not found: ID does not exist" containerID="152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"
Sep 09 19:02:50.268412 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.268348    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8"} err="failed to get container status \"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8\": rpc error: code = NotFound desc = could not find container \"152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8\": container with ID starting with 152c216d50850cf1bfdc6048989b2e73bcd28a86dcd60cf8b79e2c57d0b184f8 not found: ID does not exist"
Sep 09 19:02:50.268412 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.268370    2581 scope.go:117] "RemoveContainer" containerID="89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"
Sep 09 19:02:50.272111 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.272088    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:02:50.276395 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.276376    2581 scope.go:117] "RemoveContainer" containerID="89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"
Sep 09 19:02:50.276679 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:02:50.276662    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8\": container with ID starting with 89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8 not found: ID does not exist" containerID="89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"
Sep 09 19:02:50.276744 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.276674    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a63380-7d7bbb-44qvp"]
Sep 09 19:02:50.276744 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.276684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8"} err="failed to get container status \"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8\": rpc error: code = NotFound desc = could not find container \"89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8\": container with ID starting with 89ba674c08df2e1af43a050a1586660957de27793d17b6edfa580a94f78b53a8 not found: ID does not exist"
Sep 09 19:02:50.283326 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.283308    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311d8e02-ca7e-427c-9349-0a77be9961a1" path="/var/lib/kubelet/pods/311d8e02-ca7e-427c-9349-0a77be9961a1/volumes"
Sep 09 19:02:50.288059 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.288040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:02:50.291486 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.291469    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a63380-6ddd48c88c-77nf7"]
Sep 09 19:02:50.301551 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.301531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:02:50.309068 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:50.309048    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a63380-84994586c6-7qbgg"]
Sep 09 19:02:52.283797 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:52.283764    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" path="/var/lib/kubelet/pods/624eb4a0-fb68-474e-8dfe-d5b186c0c442/volumes"
Sep 09 19:02:52.284145 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:52.284108    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" path="/var/lib/kubelet/pods/c0efd79d-28ff-460d-b6c8-d0a2e3ac7918/volumes"
Sep 09 19:02:55.280064 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:55.279988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:02:57.171437 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:02:57.171403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:12.453716 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:12.453663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:26.360092 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:26.360058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:31.366569 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:31.366518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:32.614099 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:32.614059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:38.255057 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:38.255027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:43.876378 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:43.876335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:43.946268 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:43.946229    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:44.201546 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:44.201471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:44.883553 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:44.883521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:55.215185 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:55.215148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:55.315440 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:55.315406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:55.425190 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:55.425159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:55.532242 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:55.532160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:03:55.804303 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:03:55.804208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:01.613117 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:01.613075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:01.713871 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:01.713837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:01.860239 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:01.860206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:02.415603 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:02.415569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:12.642488 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.642452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 19:04:12.642985 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.642666    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" podUID="1c681c76-476f-4d4f-b222-0633d6b3c75d" containerName="maas-api" containerID="cri-o://93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c" gracePeriod=30
Sep 09 19:04:12.676513 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.676486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:12.750321 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.750285    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:12.813163 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.813086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:12.890392 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.890364    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 19:04:12.913207 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:12.913106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:13.013316 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.013287    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w49wb\" (UniqueName: \"kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb\") pod \"1c681c76-476f-4d4f-b222-0633d6b3c75d\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") "
Sep 09 19:04:13.013513 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.013364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") pod \"1c681c76-476f-4d4f-b222-0633d6b3c75d\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") "
Sep 09 19:04:13.013513 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.013420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs\") pod \"1c681c76-476f-4d4f-b222-0633d6b3c75d\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") "
Sep 09 19:04:13.013513 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.013487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca\") pod \"1c681c76-476f-4d4f-b222-0633d6b3c75d\" (UID: \"1c681c76-476f-4d4f-b222-0633d6b3c75d\") "
Sep 09 19:04:13.014099 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.013996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c681c76-476f-4d4f-b222-0633d6b3c75d" (UID: "1c681c76-476f-4d4f-b222-0633d6b3c75d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:13.015686 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.015657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c681c76-476f-4d4f-b222-0633d6b3c75d" (UID: "1c681c76-476f-4d4f-b222-0633d6b3c75d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:13.015825 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.015748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c681c76-476f-4d4f-b222-0633d6b3c75d" (UID: "1c681c76-476f-4d4f-b222-0633d6b3c75d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:13.015918 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.015904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb" (OuterVolumeSpecName: "kube-api-access-w49wb") pod "1c681c76-476f-4d4f-b222-0633d6b3c75d" (UID: "1c681c76-476f-4d4f-b222-0633d6b3c75d"). InnerVolumeSpecName "kube-api-access-w49wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:13.114757 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.114682    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114757 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.114745    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c681c76-476f-4d4f-b222-0633d6b3c75d-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114757 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.114759    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49wb\" (UniqueName: \"kubernetes.io/projected/1c681c76-476f-4d4f-b222-0633d6b3c75d-kube-api-access-w49wb\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114757 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.114768    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c681c76-476f-4d4f-b222-0633d6b3c75d-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.299435 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.299352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:13.634193 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.634164    2581 generic.go:356] "Generic (PLEG): container finished" podID="1c681c76-476f-4d4f-b222-0633d6b3c75d" containerID="93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c" exitCode=0
Sep 09 19:04:13.634369 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.634210    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"
Sep 09 19:04:13.634369 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.634254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" event={"ID":"1c681c76-476f-4d4f-b222-0633d6b3c75d","Type":"ContainerDied","Data":"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"}
Sep 09 19:04:13.634369 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.634308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9" event={"ID":"1c681c76-476f-4d4f-b222-0633d6b3c75d","Type":"ContainerDied","Data":"841346f576610a528125265ef7475f1708eafeddf551c9bd906cab3dda70a50d"}
Sep 09 19:04:13.634369 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.634338    2581 scope.go:117] "RemoveContainer" containerID="93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"
Sep 09 19:04:13.643592 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.643436    2581 scope.go:117] "RemoveContainer" containerID="93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"
Sep 09 19:04:13.643832 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:04:13.643739    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c\": container with ID starting with 93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c not found: ID does not exist" containerID="93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"
Sep 09 19:04:13.643832 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.643770    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c"} err="failed to get container status \"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c\": rpc error: code = NotFound desc = could not find container \"93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c\": container with ID starting with 93ad5c0ed736f42b0b31b1cfe267df6cd0785376412a0a39452ef7f9660c743c not found: ID does not exist"
Sep 09 19:04:13.655187 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.655135    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 19:04:13.656813 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:13.656791    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-282de3-f5bf487bb-lfdt9"]
Sep 09 19:04:14.283310 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:14.283274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c681c76-476f-4d4f-b222-0633d6b3c75d" path="/var/lib/kubelet/pods/1c681c76-476f-4d4f-b222-0633d6b3c75d/volumes"
Sep 09 19:04:19.236928 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:19.236878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:19.304626 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:19.304569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:19.542504 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:19.542428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:20.102293 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:20.102253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:26.803528 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:26.803450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:30.370543 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.370511    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 19:04:30.372185 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.371924    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" podUID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" containerName="maas-api" containerID="cri-o://7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c" gracePeriod=30
Sep 09 19:04:30.402994 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.402969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:30.502623 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.502597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:30.593219 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.593189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:30.618430 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.618410    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 19:04:30.652760 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.652651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:30.709556 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.709527    2581 generic.go:356] "Generic (PLEG): container finished" podID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" containerID="7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c" exitCode=0
Sep 09 19:04:30.709739 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.709572    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"
Sep 09 19:04:30.709739 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.709625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" event={"ID":"4586543d-6d8e-44fd-a13a-d94a6bbb539f","Type":"ContainerDied","Data":"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"}
Sep 09 19:04:30.709739 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.709668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs" event={"ID":"4586543d-6d8e-44fd-a13a-d94a6bbb539f","Type":"ContainerDied","Data":"196822f06b1961230b6965189e768a451516d4dd92ad631902bd15f769d14464"}
Sep 09 19:04:30.709739 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.709694    2581 scope.go:117] "RemoveContainer" containerID="7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"
Sep 09 19:04:30.718137 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.718122    2581 scope.go:117] "RemoveContainer" containerID="7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"
Sep 09 19:04:30.718416 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:04:30.718397    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c\": container with ID starting with 7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c not found: ID does not exist" containerID="7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"
Sep 09 19:04:30.718465 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.718426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c"} err="failed to get container status \"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c\": rpc error: code = NotFound desc = could not find container \"7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c\": container with ID starting with 7ad9246b3b1544189f99a3459d4459d3c4bc657698c179762f175b0dbacede3c not found: ID does not exist"
Sep 09 19:04:30.769768 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.769733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca\") pod \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") "
Sep 09 19:04:30.769926 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.769822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs\") pod \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") "
Sep 09 19:04:30.769926 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.769851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls\") pod \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") "
Sep 09 19:04:30.769926 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.769902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59sk5\" (UniqueName: \"kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5\") pod \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\" (UID: \"4586543d-6d8e-44fd-a13a-d94a6bbb539f\") "
Sep 09 19:04:30.770122 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.770100    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4586543d-6d8e-44fd-a13a-d94a6bbb539f" (UID: "4586543d-6d8e-44fd-a13a-d94a6bbb539f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:30.772248 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.772225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5" (OuterVolumeSpecName: "kube-api-access-59sk5") pod "4586543d-6d8e-44fd-a13a-d94a6bbb539f" (UID: "4586543d-6d8e-44fd-a13a-d94a6bbb539f"). InnerVolumeSpecName "kube-api-access-59sk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:30.772488 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.772473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4586543d-6d8e-44fd-a13a-d94a6bbb539f" (UID: "4586543d-6d8e-44fd-a13a-d94a6bbb539f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:30.772541 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.772502    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4586543d-6d8e-44fd-a13a-d94a6bbb539f" (UID: "4586543d-6d8e-44fd-a13a-d94a6bbb539f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:30.871110 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.871077    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.871110 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.871106    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4586543d-6d8e-44fd-a13a-d94a6bbb539f-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.871110 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.871115    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59sk5\" (UniqueName: \"kubernetes.io/projected/4586543d-6d8e-44fd-a13a-d94a6bbb539f-kube-api-access-59sk5\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.871344 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:30.871127    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4586543d-6d8e-44fd-a13a-d94a6bbb539f-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:31.011172 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:31.011091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:31.029857 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:31.029821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 19:04:31.033217 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:31.033190    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e012d-7f7bb6865f-cxtxs"]
Sep 09 19:04:32.284164 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:32.284128    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" path="/var/lib/kubelet/pods/4586543d-6d8e-44fd-a13a-d94a6bbb539f/volumes"
Sep 09 19:04:37.166147 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:37.166110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:37.242233 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:37.242204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:39.744113 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:39.744084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:04:40.393173 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.393138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 19:04:40.393350 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.393320    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" podUID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" containerName="manager" containerID="cri-o://76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308" gracePeriod=30
Sep 09 19:04:40.637568 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.637541    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 19:04:40.755236 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.755153    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" containerID="76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308" exitCode=0
Sep 09 19:04:40.755236 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.755204    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-vkbst"
Sep 09 19:04:40.755236 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.755211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" event={"ID":"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff","Type":"ContainerDied","Data":"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"}
Sep 09 19:04:40.755675 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.755254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-vkbst" event={"ID":"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff","Type":"ContainerDied","Data":"dbe9a7551c9b3ebc74299ebb77cf923225185d71783c923d6c389a8226f5e25a"}
Sep 09 19:04:40.755675 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.755275    2581 scope.go:117] "RemoveContainer" containerID="76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"
Sep 09 19:04:40.760662 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.760643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs\") pod \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") "
Sep 09 19:04:40.760809 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.760703    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m46fs\" (UniqueName: \"kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs\") pod \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") "
Sep 09 19:04:40.760809 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.760782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert\") pod \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\" (UID: \"e4aa3225-b6e9-49d8-b5e6-d15d482b61ff\") "
Sep 09 19:04:40.762984 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.762954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" (UID: "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:40.762984 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.762971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs" (OuterVolumeSpecName: "kube-api-access-m46fs") pod "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" (UID: "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff"). InnerVolumeSpecName "kube-api-access-m46fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:40.763196 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.763182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" (UID: "e4aa3225-b6e9-49d8-b5e6-d15d482b61ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:04:40.764739 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.764686    2581 scope.go:117] "RemoveContainer" containerID="76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"
Sep 09 19:04:40.765041 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:04:40.765016    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308\": container with ID starting with 76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308 not found: ID does not exist" containerID="76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"
Sep 09 19:04:40.765103 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.765052    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308"} err="failed to get container status \"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308\": rpc error: code = NotFound desc = could not find container \"76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308\": container with ID starting with 76df3aedcb40d7bb60761edcb9de4ca59f7a79812e59cf2d5606c6553f299308 not found: ID does not exist"
Sep 09 19:04:40.862051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.862024    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-webhook-cert\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:40.862051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.862047    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:40.862051 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:40.862056    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46fs\" (UniqueName: \"kubernetes.io/projected/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff-kube-api-access-m46fs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:04:41.077241 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:41.077209    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 19:04:41.081190 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:41.081163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-vkbst"]
Sep 09 19:04:42.289742 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:04:42.289680    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" path="/var/lib/kubelet/pods/e4aa3225-b6e9-49d8-b5e6-d15d482b61ff/volumes"
Sep 09 19:05:41.931578 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931544    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931873    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931898    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931907    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05886091-d1ee-435b-8390-c78bb91ef18b" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931912    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="05886091-d1ee-435b-8390-c78bb91ef18b" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931920    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d8e02-ca7e-427c-9349-0a77be9961a1" containerName="payload-pre-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931926    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d8e02-ca7e-427c-9349-0a77be9961a1" containerName="payload-pre-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931934    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" containerName="manager"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" containerName="manager"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931945    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c681c76-476f-4d4f-b222-0633d6b3c75d" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931950    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c681c76-476f-4d4f-b222-0633d6b3c75d" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931956    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931970    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.931976    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932028    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0efd79d-28ff-460d-b6c8-d0a2e3ac7918" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932035    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a2cd66-10d9-404e-9e14-bd0d905e67b3" containerName="payload-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932040    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="311d8e02-ca7e-427c-9349-0a77be9961a1" containerName="payload-pre-processing"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932047    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="624eb4a0-fb68-474e-8dfe-d5b186c0c442" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932051    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c681c76-476f-4d4f-b222-0633d6b3c75d" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="05886091-d1ee-435b-8390-c78bb91ef18b" containerName="maas-api"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932063    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4aa3225-b6e9-49d8-b5e6-d15d482b61ff" containerName="manager"
Sep 09 19:05:41.932081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.932068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4586543d-6d8e-44fd-a13a-d94a6bbb539f" containerName="maas-api"
Sep 09 19:05:41.934900 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.934884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:41.938686 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.938664    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 19:05:41.938826 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.938662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 19:05:41.938826 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.938662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 19:05:41.938826 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.938813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ns67g\""
Sep 09 19:05:41.946325 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.946305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:05:41.972392 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.972358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:41.972525 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.972397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d2c\" (UniqueName: \"kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:41.972525 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:41.972422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.073148 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.073114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.073326 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.073170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d2c\" (UniqueName: \"kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.073326 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.073200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.075843 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.075818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.075984 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.075865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.081394 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.081369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d2c\" (UniqueName: \"kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c\") pod \"maas-controller-bbb54fb6c-zqcg8\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") " pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.245643 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.245554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:42.530958 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.530933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:05:42.532919 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:05:42.532888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b723bc_e103_4930_afb8_afbc568a7ff9.slice/crio-3cddb7a4d952356fc4fe891d3376692444c1c75c6d333f424fee154253bbfb1f WatchSource:0}: Error finding container 3cddb7a4d952356fc4fe891d3376692444c1c75c6d333f424fee154253bbfb1f: Status 404 returned error can't find the container with id 3cddb7a4d952356fc4fe891d3376692444c1c75c6d333f424fee154253bbfb1f
Sep 09 19:05:42.534314 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:42.534295    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:05:43.021367 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:43.021326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" event={"ID":"b1b723bc-e103-4930-afb8-afbc568a7ff9","Type":"ContainerStarted","Data":"3cddb7a4d952356fc4fe891d3376692444c1c75c6d333f424fee154253bbfb1f"}
Sep 09 19:05:44.026903 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:44.026861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" event={"ID":"b1b723bc-e103-4930-afb8-afbc568a7ff9","Type":"ContainerStarted","Data":"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"}
Sep 09 19:05:44.027284 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:44.027029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:05:44.044960 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:44.044916    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" podStartSLOduration=2.546955328 podStartE2EDuration="3.044899907s" totalImagesPullingTime="497.944579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:05:44.043988693 +0000 UTC m=+2050.275595055" watchObservedRunningTime="2026-09-09 19:05:44.044899907 +0000 UTC m=+2050.276506269"
Sep 09 19:05:55.034136 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:05:55.034060    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:06:01.481932 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:01.481882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:06:02.220661 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.220633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:06:02.325531 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.325496    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:06:02.325771 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.325744    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" podUID="b1b723bc-e103-4930-afb8-afbc568a7ff9" containerName="manager" containerID="cri-o://d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c" gracePeriod=30
Sep 09 19:06:02.574573 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.574546    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:06:02.641532 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.641496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert\") pod \"b1b723bc-e103-4930-afb8-afbc568a7ff9\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") "
Sep 09 19:06:02.641747 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.641541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7d2c\" (UniqueName: \"kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c\") pod \"b1b723bc-e103-4930-afb8-afbc568a7ff9\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") "
Sep 09 19:06:02.641747 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.641576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs\") pod \"b1b723bc-e103-4930-afb8-afbc568a7ff9\" (UID: \"b1b723bc-e103-4930-afb8-afbc568a7ff9\") "
Sep 09 19:06:02.643868 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.643834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1b723bc-e103-4930-afb8-afbc568a7ff9" (UID: "b1b723bc-e103-4930-afb8-afbc568a7ff9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:06:02.643982 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.643883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c" (OuterVolumeSpecName: "kube-api-access-s7d2c") pod "b1b723bc-e103-4930-afb8-afbc568a7ff9" (UID: "b1b723bc-e103-4930-afb8-afbc568a7ff9"). InnerVolumeSpecName "kube-api-access-s7d2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:06:02.644081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.644062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b1b723bc-e103-4930-afb8-afbc568a7ff9" (UID: "b1b723bc-e103-4930-afb8-afbc568a7ff9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:06:02.742470 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.742428    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-webhook-cert\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:06:02.742470 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.742461    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7d2c\" (UniqueName: \"kubernetes.io/projected/b1b723bc-e103-4930-afb8-afbc568a7ff9-kube-api-access-s7d2c\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:06:02.742470 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:02.742474    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b723bc-e103-4930-afb8-afbc568a7ff9-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:06:03.111440 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.111399    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1b723bc-e103-4930-afb8-afbc568a7ff9" containerID="d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c" exitCode=0
Sep 09 19:06:03.111650 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.111446    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8"
Sep 09 19:06:03.111650 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.111490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" event={"ID":"b1b723bc-e103-4930-afb8-afbc568a7ff9","Type":"ContainerDied","Data":"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"}
Sep 09 19:06:03.111650 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.111551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-zqcg8" event={"ID":"b1b723bc-e103-4930-afb8-afbc568a7ff9","Type":"ContainerDied","Data":"3cddb7a4d952356fc4fe891d3376692444c1c75c6d333f424fee154253bbfb1f"}
Sep 09 19:06:03.111650 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.111585    2581 scope.go:117] "RemoveContainer" containerID="d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"
Sep 09 19:06:03.126957 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.126935    2581 scope.go:117] "RemoveContainer" containerID="d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"
Sep 09 19:06:03.127292 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:06:03.127271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c\": container with ID starting with d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c not found: ID does not exist" containerID="d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"
Sep 09 19:06:03.127336 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.127300    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c"} err="failed to get container status \"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c\": rpc error: code = NotFound desc = could not find container \"d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c\": container with ID starting with d9890c5196d7ddc7a01b67cfca7763b6b10c4e4684350bf81f4e611634675d0c not found: ID does not exist"
Sep 09 19:06:03.138379 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.138350    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:06:03.141686 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.141662    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-zqcg8"]
Sep 09 19:06:03.867763 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.867702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:06:03.974433 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.974405    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-96c7p"]
Sep 09 19:06:03.974777 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.974763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b723bc-e103-4930-afb8-afbc568a7ff9" containerName="manager"
Sep 09 19:06:03.974835 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.974779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b723bc-e103-4930-afb8-afbc568a7ff9" containerName="manager"
Sep 09 19:06:03.974875 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.974848    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b723bc-e103-4930-afb8-afbc568a7ff9" containerName="manager"
Sep 09 19:06:03.979145 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.979124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:03.981635 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.981615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 19:06:03.982795 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.982778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 19:06:03.983052 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.983031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ns67g\""
Sep 09 19:06:03.983115 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.983057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 19:06:03.987093 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:03.987058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-96c7p"]
Sep 09 19:06:04.054656 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.054615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-metrics-certs\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.054858 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.054691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-webhook-cert\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.054858 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.054726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg22c\" (UniqueName: \"kubernetes.io/projected/02dcae4f-2697-4a38-8375-fd4ef4044ce1-kube-api-access-fg22c\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.155415 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.155315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-webhook-cert\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.155415 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.155359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg22c\" (UniqueName: \"kubernetes.io/projected/02dcae4f-2697-4a38-8375-fd4ef4044ce1-kube-api-access-fg22c\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.155415 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.155405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-metrics-certs\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.157997 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.157968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-webhook-cert\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.158101 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.158080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02dcae4f-2697-4a38-8375-fd4ef4044ce1-metrics-certs\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.164403 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.164379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg22c\" (UniqueName: \"kubernetes.io/projected/02dcae4f-2697-4a38-8375-fd4ef4044ce1-kube-api-access-fg22c\") pod \"maas-controller-bbb54fb6c-96c7p\" (UID: \"02dcae4f-2697-4a38-8375-fd4ef4044ce1\") " pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.284385 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.284348    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b723bc-e103-4930-afb8-afbc568a7ff9" path="/var/lib/kubelet/pods/b1b723bc-e103-4930-afb8-afbc568a7ff9/volumes"
Sep 09 19:06:04.289639 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.289618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:04.561576 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:04.561549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbb54fb6c-96c7p"]
Sep 09 19:06:04.563984 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:06:04.563958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dcae4f_2697_4a38_8375_fd4ef4044ce1.slice/crio-744e25e1806306e18acf8123907e430dbf76c524e6f357e203296e8d067744c2 WatchSource:0}: Error finding container 744e25e1806306e18acf8123907e430dbf76c524e6f357e203296e8d067744c2: Status 404 returned error can't find the container with id 744e25e1806306e18acf8123907e430dbf76c524e6f357e203296e8d067744c2
Sep 09 19:06:05.122318 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:05.122286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-96c7p" event={"ID":"02dcae4f-2697-4a38-8375-fd4ef4044ce1","Type":"ContainerStarted","Data":"bf347ad5b7a480356e5ddd084e45c129c681142751f0367137dbdc824d9df107"}
Sep 09 19:06:05.122686 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:05.122343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbb54fb6c-96c7p" event={"ID":"02dcae4f-2697-4a38-8375-fd4ef4044ce1","Type":"ContainerStarted","Data":"744e25e1806306e18acf8123907e430dbf76c524e6f357e203296e8d067744c2"}
Sep 09 19:06:05.122686 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:05.122460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:05.140685 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:05.140629    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbb54fb6c-96c7p" podStartSLOduration=1.735309274 podStartE2EDuration="2.140613107s" totalImagesPullingTime="405.303833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:06:05.140017849 +0000 UTC m=+2071.371624227" watchObservedRunningTime="2026-09-09 19:06:05.140613107 +0000 UTC m=+2071.372219469"
Sep 09 19:06:16.129091 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:16.129057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbb54fb6c-96c7p"
Sep 09 19:06:59.226894 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:06:59.226860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:08:41.625993 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:08:41.625947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:08:52.974615 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:08:52.974579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:09:10.299878 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:10.299819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:09:10.904424 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:10.904393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:09:37.883506 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.883473    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"]
Sep 09 19:09:37.886530 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.886514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:37.900944 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.900836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"]
Sep 09 19:09:37.970559 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.970526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-metrics-certs\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:37.970752 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.970571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzz2\" (UniqueName: \"kubernetes.io/projected/2f108dfd-444d-459a-8077-ba528175505a-kube-api-access-drzz2\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:37.970752 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.970594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f108dfd-444d-459a-8077-ba528175505a-openshift-service-ca\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:37.970752 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:37.970611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-maas-api-tls\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.071990 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.071956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-metrics-certs\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.072170 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.072033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzz2\" (UniqueName: \"kubernetes.io/projected/2f108dfd-444d-459a-8077-ba528175505a-kube-api-access-drzz2\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.072170 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.072063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f108dfd-444d-459a-8077-ba528175505a-openshift-service-ca\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.072170 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.072094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-maas-api-tls\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.072903 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.072879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f108dfd-444d-459a-8077-ba528175505a-openshift-service-ca\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.074606 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.074582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-maas-api-tls\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.074758 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.074739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f108dfd-444d-459a-8077-ba528175505a-metrics-certs\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.080958 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.080931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzz2\" (UniqueName: \"kubernetes.io/projected/2f108dfd-444d-459a-8077-ba528175505a-kube-api-access-drzz2\") pod \"maas-api-6d75774dd5-d8xm4\" (UID: \"2f108dfd-444d-459a-8077-ba528175505a\") " pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.198426 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.198334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:38.472370 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:38.472345    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"]
Sep 09 19:09:38.474117 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:09:38.474093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f108dfd_444d_459a_8077_ba528175505a.slice/crio-1097cd6f494f9945cb06f7daca4f1528c9fddfb4775119ed40aae05ebeb5d838 WatchSource:0}: Error finding container 1097cd6f494f9945cb06f7daca4f1528c9fddfb4775119ed40aae05ebeb5d838: Status 404 returned error can't find the container with id 1097cd6f494f9945cb06f7daca4f1528c9fddfb4775119ed40aae05ebeb5d838
Sep 09 19:09:39.046247 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:39.046216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4" event={"ID":"2f108dfd-444d-459a-8077-ba528175505a","Type":"ContainerStarted","Data":"2854d99cf4e3f6f93765481f144a02d66d85f0cd2b75e216704a9009f550f85e"}
Sep 09 19:09:39.046651 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:39.046270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4" event={"ID":"2f108dfd-444d-459a-8077-ba528175505a","Type":"ContainerStarted","Data":"1097cd6f494f9945cb06f7daca4f1528c9fddfb4775119ed40aae05ebeb5d838"}
Sep 09 19:09:39.046651 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:39.046374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:39.066489 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:39.066444    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4" podStartSLOduration=1.647898465 podStartE2EDuration="2.06642976s" totalImagesPullingTime="418.531295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:09:39.064295787 +0000 UTC m=+2285.295902141" watchObservedRunningTime="2026-09-09 19:09:39.06642976 +0000 UTC m=+2285.298036122"
Sep 09 19:09:45.056288 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:45.056260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d75774dd5-d8xm4"
Sep 09 19:09:45.120275 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:45.120245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 19:09:45.120489 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:45.120463    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" podUID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" containerName="maas-api" containerID="cri-o://88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75" gracePeriod=30
Sep 09 19:09:46.361772 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.361747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 19:09:46.447484 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.447404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt49r\" (UniqueName: \"kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r\") pod \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") "
Sep 09 19:09:46.447484 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.447462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs\") pod \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") "
Sep 09 19:09:46.447748 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.447496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca\") pod \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") "
Sep 09 19:09:46.447748 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.447522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls\") pod \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\" (UID: \"6d1a9018-5d2b-4352-b462-6f9bb50b1d13\") "
Sep 09 19:09:46.447919 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.447883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d1a9018-5d2b-4352-b462-6f9bb50b1d13" (UID: "6d1a9018-5d2b-4352-b462-6f9bb50b1d13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:09:46.449860 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.449834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r" (OuterVolumeSpecName: "kube-api-access-lt49r") pod "6d1a9018-5d2b-4352-b462-6f9bb50b1d13" (UID: "6d1a9018-5d2b-4352-b462-6f9bb50b1d13"). InnerVolumeSpecName "kube-api-access-lt49r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:09:46.449860 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.449847    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d1a9018-5d2b-4352-b462-6f9bb50b1d13" (UID: "6d1a9018-5d2b-4352-b462-6f9bb50b1d13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:09:46.449996 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.449861    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d1a9018-5d2b-4352-b462-6f9bb50b1d13" (UID: "6d1a9018-5d2b-4352-b462-6f9bb50b1d13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:09:46.549041 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.549009    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-metrics-certs\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:09:46.549041 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.549034    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-openshift-service-ca\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:09:46.549041 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.549044    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-maas-api-tls\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:09:46.549259 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:46.549052    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt49r\" (UniqueName: \"kubernetes.io/projected/6d1a9018-5d2b-4352-b462-6f9bb50b1d13-kube-api-access-lt49r\") on node \"ip-10-0-135-104.ec2.internal\" DevicePath \"\""
Sep 09 19:09:47.081785 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.081745    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" containerID="88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75" exitCode=0
Sep 09 19:09:47.081785 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.081790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" event={"ID":"6d1a9018-5d2b-4352-b462-6f9bb50b1d13","Type":"ContainerDied","Data":"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"}
Sep 09 19:09:47.081999 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.081800    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"
Sep 09 19:09:47.081999 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.081835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj" event={"ID":"6d1a9018-5d2b-4352-b462-6f9bb50b1d13","Type":"ContainerDied","Data":"1f51fb2d51b5612786faefd2a395fc71a764047a25df00500e7a25d357f27980"}
Sep 09 19:09:47.081999 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.081862    2581 scope.go:117] "RemoveContainer" containerID="88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"
Sep 09 19:09:47.091124 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.091105    2581 scope.go:117] "RemoveContainer" containerID="88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"
Sep 09 19:09:47.091419 ip-10-0-135-104 kubenswrapper[2581]: E0909 19:09:47.091402    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75\": container with ID starting with 88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75 not found: ID does not exist" containerID="88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"
Sep 09 19:09:47.091483 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.091427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75"} err="failed to get container status \"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75\": rpc error: code = NotFound desc = could not find container \"88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75\": container with ID starting with 88389fcd9814e320ef1e9277f1e81c08452ac484022683a09164391053f86f75 not found: ID does not exist"
Sep 09 19:09:47.103763 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.103736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 19:09:47.108863 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:47.108843    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb87fb765-gmhcj"]
Sep 09 19:09:48.284382 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:09:48.284347    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" path="/var/lib/kubelet/pods/6d1a9018-5d2b-4352-b462-6f9bb50b1d13/volumes"
Sep 09 19:10:05.867456 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:05.867423    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t9lkz_2686874d-43c6-4140-9f7d-fbe9dbd5ddb9/manager/0.log"
Sep 09 19:10:06.002808 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:06.002754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fxnx2_a09abe68-df91-42b1-9c06-eafb7fac39b9/manager/0.log"
Sep 09 19:10:06.131445 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:06.131347    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbb54fb6c-96c7p_02dcae4f-2697-4a38-8375-fd4ef4044ce1/manager/0.log"
Sep 09 19:10:06.414073 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:06.413985    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-94m4h_8cb9ee65-0c59-4934-9bff-98575f73271d/manager/0.log"
Sep 09 19:10:06.534195 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:06.534169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-mmsvq_736fa91d-a021-450a-8b82-52594a11ce3b/copy-manifests/0.log"
Sep 09 19:10:06.541654 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:06.541624    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-mmsvq_736fa91d-a021-450a-8b82-52594a11ce3b/manager/0.log"
Sep 09 19:10:07.191299 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:07.191270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d75774dd5-d8xm4_2f108dfd-444d-459a-8077-ba528175505a/maas-api/0.log"
Sep 09 19:10:07.563341 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:07.563263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-n69sd_e7c0cf24-ef74-492e-b12c-2a186a8e11f2/postgres/0.log"
Sep 09 19:10:08.522042 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:08.522012    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-z7nss_760ec79e-6761-49ba-a4ed-71032df78b0b/manager/0.log"
Sep 09 19:10:09.644575 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:09.644541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r68j7_00c1a572-3c6a-4ecc-9dc8-e920942892ce/discovery/0.log"
Sep 09 19:10:09.767951 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:09.767901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-2phn9_246d7540-c8fb-4f28-b6cd-b42be3c2539e/kube-auth-proxy/0.log"
Sep 09 19:10:10.156814 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:10.156784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wfthn_7c3a4a98-65ff-4b54-a949-92ba038e691a/payload-pre-processing/0.log"
Sep 09 19:10:10.908498 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:10.908463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dz7vc_f309db75-d227-447d-bab3-548d1b077896/main/0.log"
Sep 09 19:10:11.027015 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:11.026983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vbqgk_20ad08f4-4325-4fa0-a92f-a27d8995cd34/main/0.log"
Sep 09 19:10:11.378927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:11.378900    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-d59vw_2f06fc30-622b-4eff-ae24-57e283604c2a/main/0.log"
Sep 09 19:10:17.361422 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:17.361382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:10:17.515371 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:17.515338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log" limit=1024
Sep 09 19:10:24.144818 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:24.144784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9hhwv_c45724fc-0e25-4804-8f07-891fb9b57ebf/global-pull-secret-syncer/0.log"
Sep 09 19:10:24.287236 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:24.287210    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5284w_aaab7097-1205-4494-aa26-1c55db0a57bc/konnectivity-agent/0.log"
Sep 09 19:10:24.439120 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:24.439041    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-104.ec2.internal_9f59d360c99ceabc7a2e0508d7e2b003/haproxy/0.log"
Sep 09 19:10:28.997075 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:28.996988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-z7nss_760ec79e-6761-49ba-a4ed-71032df78b0b/manager/0.log"
Sep 09 19:10:31.294572 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:31.294542    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/node-exporter/0.log"
Sep 09 19:10:31.317102 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:31.317073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/kube-rbac-proxy/0.log"
Sep 09 19:10:31.353182 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:31.353161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rm4hn_2683c1ed-ef49-49b5-a980-7c940654fce1/init-textfile/0.log"
Sep 09 19:10:34.186272 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.186233    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"]
Sep 09 19:10:34.186764 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.186745    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" containerName="maas-api"
Sep 09 19:10:34.186847 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.186765    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" containerName="maas-api"
Sep 09 19:10:34.186901 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.186881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d1a9018-5d2b-4352-b462-6f9bb50b1d13" containerName="maas-api"
Sep 09 19:10:34.189821 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.189800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.196175 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.196154    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dz2cr\"/\"kube-root-ca.crt\""
Sep 09 19:10:34.197239 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.197223    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dz2cr\"/\"openshift-service-ca.crt\""
Sep 09 19:10:34.197315 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.197246    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dz2cr\"/\"default-dockercfg-dnjv8\""
Sep 09 19:10:34.208755 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.208733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"]
Sep 09 19:10:34.247281 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.247247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-podres\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.247420 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.247305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-sys\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.247420 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.247356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7st\" (UniqueName: \"kubernetes.io/projected/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-kube-api-access-9x7st\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.247496 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.247425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-proc\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.247496 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.247463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-lib-modules\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348736 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-podres\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-sys\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7st\" (UniqueName: \"kubernetes.io/projected/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-kube-api-access-9x7st\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-podres\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-proc\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-sys\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-proc\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.348927 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.348915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-lib-modules\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.349204 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.349012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-lib-modules\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.361289 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.361266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7st\" (UniqueName: \"kubernetes.io/projected/2eabb334-3cc9-4e87-b0b6-1ad0b46e1782-kube-api-access-9x7st\") pod \"perf-node-gather-daemonset-cg268\" (UID: \"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.499577 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.499498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:34.783914 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:34.783860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"]
Sep 09 19:10:34.786788 ip-10-0-135-104 kubenswrapper[2581]: W0909 19:10:34.786762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2eabb334_3cc9_4e87_b0b6_1ad0b46e1782.slice/crio-06a9fa04770c8ff9da6c17f570d94301bf15d7f53a5f8d4d3fdd4690952cafdc WatchSource:0}: Error finding container 06a9fa04770c8ff9da6c17f570d94301bf15d7f53a5f8d4d3fdd4690952cafdc: Status 404 returned error can't find the container with id 06a9fa04770c8ff9da6c17f570d94301bf15d7f53a5f8d4d3fdd4690952cafdc
Sep 09 19:10:35.290320 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:35.290285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268" event={"ID":"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782","Type":"ContainerStarted","Data":"5a007d743f418e4212ff3165d727f33c64cadf62755a7e8b548b9630022623d3"}
Sep 09 19:10:35.290783 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:35.290336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268" event={"ID":"2eabb334-3cc9-4e87-b0b6-1ad0b46e1782","Type":"ContainerStarted","Data":"06a9fa04770c8ff9da6c17f570d94301bf15d7f53a5f8d4d3fdd4690952cafdc"}
Sep 09 19:10:35.290783 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:35.290447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:35.313612 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:35.313565    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268" podStartSLOduration=1.313549802 podStartE2EDuration="1.313549802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:10:35.313122893 +0000 UTC m=+2341.544729255" watchObservedRunningTime="2026-09-09 19:10:35.313549802 +0000 UTC m=+2341.545156197"
Sep 09 19:10:37.269661 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:37.269628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vf9rh_0f3173bc-3ab6-4b3b-889f-8c95808239f7/dns/0.log"
Sep 09 19:10:37.301118 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:37.301090    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vf9rh_0f3173bc-3ab6-4b3b-889f-8c95808239f7/kube-rbac-proxy/0.log"
Sep 09 19:10:37.338277 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:37.338246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h868k_549f102e-c718-4cdb-8a92-392846408a79/dns-node-resolver/0.log"
Sep 09 19:10:38.011795 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:38.011766    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2b2wp_9648bc4f-28e2-4501-b9d7-f8d04caa8550/node-ca/0.log"
Sep 09 19:10:39.475642 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:39.475616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r68j7_00c1a572-3c6a-4ecc-9dc8-e920942892ce/discovery/0.log"
Sep 09 19:10:39.512587 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:39.512559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-2phn9_246d7540-c8fb-4f28-b6cd-b42be3c2539e/kube-auth-proxy/0.log"
Sep 09 19:10:39.650257 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:39.650231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wfthn_7c3a4a98-65ff-4b54-a949-92ba038e691a/payload-pre-processing/0.log"
Sep 09 19:10:40.676886 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:40.676826    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qm8n4_80737e9b-b5fe-4d8e-a8e3-d8ee4984fee3/serve-healthcheck-canary/0.log"
Sep 09 19:10:41.305081 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:41.305050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cg268"
Sep 09 19:10:41.464655 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:41.464619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czlqj_96041057-1b11-44da-b6ba-21c0425d7b3f/kube-rbac-proxy/0.log"
Sep 09 19:10:41.498481 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:41.498457    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czlqj_96041057-1b11-44da-b6ba-21c0425d7b3f/exporter/0.log"
Sep 09 19:10:41.525375 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:41.525350    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czlqj_96041057-1b11-44da-b6ba-21c0425d7b3f/extractor/0.log"
Sep 09 19:10:44.103352 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.103323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t9lkz_2686874d-43c6-4140-9f7d-fbe9dbd5ddb9/manager/0.log"
Sep 09 19:10:44.426957 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.426924    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fxnx2_a09abe68-df91-42b1-9c06-eafb7fac39b9/manager/0.log"
Sep 09 19:10:44.556774 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.556738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbb54fb6c-96c7p_02dcae4f-2697-4a38-8375-fd4ef4044ce1/manager/0.log"
Sep 09 19:10:44.862898 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.862869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-94m4h_8cb9ee65-0c59-4934-9bff-98575f73271d/manager/0.log"
Sep 09 19:10:44.926388 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.926358    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r68j7_00c1a572-3c6a-4ecc-9dc8-e920942892ce/discovery/0.log"
Sep 09 19:10:44.985045 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.984999    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-2phn9_246d7540-c8fb-4f28-b6cd-b42be3c2539e/kube-auth-proxy/0.log"
Sep 09 19:10:44.994782 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:44.994751    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-mmsvq_736fa91d-a021-450a-8b82-52594a11ce3b/manager/0.log"
Sep 09 19:10:45.026838 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:45.026815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-mmsvq_736fa91d-a021-450a-8b82-52594a11ce3b/copy-manifests/0.log"
Sep 09 19:10:45.074222 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:45.074198    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wfthn_7c3a4a98-65ff-4b54-a949-92ba038e691a/payload-pre-processing/0.log"
Sep 09 19:10:46.723449 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:46.723414    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5m9kb_2a1ec764-8cfc-4b78-88dc-979542a5baa8/openshift-lws-operator/0.log"
Sep 09 19:10:47.216169 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:47.216137    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r68j7_00c1a572-3c6a-4ecc-9dc8-e920942892ce/discovery/0.log"
Sep 09 19:10:47.360413 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:47.360391    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-2phn9_246d7540-c8fb-4f28-b6cd-b42be3c2539e/kube-auth-proxy/0.log"
Sep 09 19:10:47.459086 ip-10-0-135-104 kubenswrapper[2581]: I0909 19:10:47.459055    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wfthn_7c3a4a98-65ff-4b54-a949-92ba038e691a/payload-pre-processing/0.log"